[PATCH 0/4] Allow shallow git clones

[PATCH 0/4] Allow shallow git clones

[Chase Maupin]

All,

This patch set allows for creating shallow git clones which is useful
to reduct the size of the packages git repository.  In my testing using
the git-native version 1.9.0 found in oe-core this reduced the kernel
sources size from 1.5GB to 762MB.  This issue originally came up because
there is a 2GB file size limit for InstallBuilder and this large git
history was causing the SDK to be too big.  No wthe SDK size has
reduced to something more reasonable like 1.6GB total.

The full history can still be fetched using "git pull" or "git pull --unshallow"
depending on the git version being used.

Chase Maupin (4):
  sourceipk: allow packaging shallow git clone
  arago-source-ipk: do a shallow clone of the kernel
  arago: unset ASSUME_PROVIDED for git-native
  linux-ti-staging: update bbappend for 3.14 version

 meta-arago-distro/classes/sourceipk.bbclass        |  100 +++++++++++++++++---
 .../conf/distro/arago-source-ipk.conf              |    1 +
 meta-arago-distro/conf/distro/arago.conf           |    5 +
 ....12.bbappend => linux-ti-staging_3.14.bbappend} |    2 +
 4 files changed, 97 insertions(+), 11 deletions(-)
 copy meta-arago-distro/recipes-kernel/linux/{linux-ti-staging_3.12.bbappend => linux-ti-staging_3.14.bbappend} (94%)

-- 
1.7.9.5

[PATCH 1/4] sourceipk: allow packaging shallow git clone

[Chase Maupin]

* Add another flag SRCIPK_SHALLOW_CLONE that will do a shallow
  clone of depth 1 to reduce the size of the git history.
    * The full history can be fetched again using either of the
      following commands based on the version of git being used
        - git pull --unshallow (New git versions)
        - git pull (older git versions)
* Clean up some of the variable names to make it more clear what
  they are storing.
* Create a generic function to determine the fetch reporitory

Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
---
 meta-arago-distro/classes/sourceipk.bbclass |  100 ++++++++++++++++++++++++---
 1 file changed, 89 insertions(+), 11 deletions(-)

diff --git a/meta-arago-distro/classes/sourceipk.bbclass b/meta-arago-distro/classes/sourceipk.bbclass
index 20dda76..33878ac 100644
--- a/meta-arago-distro/classes/sourceipk.bbclass
+++ b/meta-arago-distro/classes/sourceipk.bbclass
@@ -61,32 +61,111 @@ SRCIPK_INCLUDE_EXTRAFILES ?= "1"
 
 SRCIPK_PRESERVE_GIT ?= "false"
 
-adjust_git() {
+# Create a shallow clone of the git repository to reduce the size of
+# the sourceipk
+SRCIPK_SHALLOW_CLONE ?= "false"
+
+# This function will return the fetch URL for a git repository passed as
+# the first parameter.
+get_remote() {
+    git_repo="$1"
+
+    if [ "$git_repo" == "" ]
+    then
+        echo "git_repo not passed to get_remote"
+        exit 1
+    fi
+
+    cd $git_repo
+
+    # Get the remote repository fetch URL
+    remote=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
+
+    # Since the echo'ed value of this statment is the returned value redirect
+    # the output of this command to /dev/null
+    cd - > /dev/null
+
+    # echo back the remote repository URL as the output of this function
+    echo $remote
+
+    return 0
+}
+
+# Some git repositories are very large and we do not want to ship the
+# full history.  Instead we want to limit history to reduce the size while
+# still keeping the git repository in place.  The full history can be
+# fetched using git pull --unshallow or just git pull
+# NOTE: This function depends on a git version >= 1.7.10.  It will work
+#       with older versions but the size will be larger because rather
+#       than just a single branch the limited history will be a depth of
+#       1 for all branches and tags.
+limit_git_history() {
+    # By default limit the history to 1 commit since the user can always
+    # use git pull --unshallow to fetch the rest of history.  The depth
+    # level of 1 is set to keep from tracking through all merges and
+    # pulling excess history
+    commits="1"
+
+    # Temporary directory to make shallow clones in
+    gitshallowclone="${WORKDIR}/temp-git-shallow-clone"
+
+    # Change directory to the git repository to be safe
+    cd $tmp_dir/${SRCIPK_INSTALL_DIR}
+
+    # Create a temporary directory to hold the shallow git clone
+    mkdir -p $gitshallowclone
 
+    remote=`get_remote $PWD`
+
+    git clone --depth $commits --branch ${BRANCH} file://$remote $gitshallowclone
+
+    # remove original kernel clone since we will replace it with the shallow
+    # clone
+    rm -rf $tmp_dir/${SRCIPK_INSTALL_DIR}/.git
+
+    # replace the original kernel git data with the shallow clone git data
+    mv $gitshallowclone/.git $tmp_dir/${SRCIPK_INSTALL_DIR}/
+    rm -rf $gitshallowclone
+
+    cd -
+}
+
+adjust_git() {
     orig_dir="$PWD"
 
     cd $tmp_dir/${SRCIPK_INSTALL_DIR}
 
     if [ -d ".git" ]
     then
+        # Get the location of the local repository
+        local_repo=`get_remote $PWD`
 
-        # Grab path to cloned local repository
-        old=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
-
-        if [ -d $old -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
+        if [ -d $local_repo -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
         then
-            cd $old
+            cd $local_repo
+
+            # If SRCIPK_SHALLOW_CLONE is true then make a shallow copy of the
+            # git repository and then fix up the git URLs
+            if [ "${SRCIPK_SHALLOW_CLONE}" == "true" ]
+            then
+                limit_git_history
+            fi
 
             # Grab actual url used to create the repository
-            orig=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
+            remote_repo=`get_remote $PWD`
 
             cd -
 
-            git remote set-url origin $orig $old
+            git remote set-url origin $remote_repo $local_repo
 
-            # Repackage the repository so its a proper clone of the original (remote) git repository
+            # Repackage the repository so its a proper clone of the original
+            # (remote) git repository
             git repack -a -d
-            rm .git/objects/info/alternates
+
+            if [ -e .git/objects/info/alternates ]
+            then
+                rm .git/objects/info/alternates
+            fi
 
         else
             rm -rf .git
@@ -95,7 +174,6 @@ adjust_git() {
     fi
 
     cd $orig_dir
-
 }
 
 # Create a README file that describes the contents of the source ipk
-- 
1.7.9.5

[PATCH 2/4] arago-source-ipk: do a shallow clone of the kernel

[Chase Maupin]

* Without this shallow clone the kernel sources is 1.5GB. With
  a shallow clone the kernel sources are 762MB which is much
  more reasonable.

Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
---
 .../conf/distro/arago-source-ipk.conf              |    1 +
 1 file changed, 1 insertion(+)

diff --git a/meta-arago-distro/conf/distro/arago-source-ipk.conf b/meta-arago-distro/conf/distro/arago-source-ipk.conf
index f6b682d..ccb6e2d 100644
--- a/meta-arago-distro/conf/distro/arago-source-ipk.conf
+++ b/meta-arago-distro/conf/distro/arago-source-ipk.conf
@@ -152,6 +152,7 @@ CREATE_SRCIPK_pn-linux-ti-staging = "1"
 SRCIPK_INSTALL_DIR_pn-linux-ti-staging = "board-support/linux-${PV}${KERNEL_LOCALVERSION}"
 SRCIPK_PACKAGE_ARCH_pn-linux-ti-staging = "${MACHINE_ARCH}"
 SRCIPK_PRESERVE_GIT_pn-linux-ti-staging = "true"
+SRCIPK_SHALLOW_CLONE_pn-linux-ti-staging = "true"
 
 CREATE_SRCIPK_pn-linux-keystone = "1"
 SRCIPK_INSTALL_DIR_pn-linux-keystone = "board-support/linux-${PV}"
-- 
1.7.9.5

[PATCH 3/4] arago: unset ASSUME_PROVIDED for git-native

[Chase Maupin]

* To get maximum size reduction use the later version of git
  that is provided by git-native.  This reduces the kernel sources
  size from 1002MB to 762MB.
* Older versions of git work but they provide a depth of 1 for
  all branches and tags instead of just for the branch requested.
* This was tested with git version 1.7.9.5 and 1.9 as comparison
  points.

Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
---
 meta-arago-distro/conf/distro/arago.conf |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta-arago-distro/conf/distro/arago.conf b/meta-arago-distro/conf/distro/arago.conf
index b9af13f..1470f22 100644
--- a/meta-arago-distro/conf/distro/arago.conf
+++ b/meta-arago-distro/conf/distro/arago.conf
@@ -50,6 +50,11 @@ require conf/distro/include/arago-prefs.inc
 # Hack out sgx flag for now, as breaking on kernels >= 3.14
 MACHINE_FEATURES := "${@filter_out('sgx','${MACHINE_FEATURES}',d)}"
 
+# Remove the ASSUME_PROVIDED setting for git-native so that we can
+# use the latest version for the sourceipk which leads to a smaller
+# git history.
+ASSUME_PROVIDED := "${@filter_out('git-native','${ASSUME_PROVIDED}',d)}"
+
 # Allow branding on top of Arago Distro and Core TI-SDK
 # If ARAGO_BRAND is not set in local.conf, default to core
 ARAGO_BRAND ??= "core"
-- 
1.7.9.5

[PATCH 4/4] linux-ti-staging: update bbappend for 3.14 version

[Chase Maupin]

* Update the bbappend for the linux-ti-staging to work with the
  3.14 kernel version.
* Set the kernel to AUTOREV while developing the latest 3.14 kernel

Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
---
 ....12.bbappend => linux-ti-staging_3.14.bbappend} |    2 ++
 1 file changed, 2 insertions(+)
 copy meta-arago-distro/recipes-kernel/linux/{linux-ti-staging_3.12.bbappend => linux-ti-staging_3.14.bbappend} (94%)

diff --git a/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.12.bbappend b/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.14.bbappend
similarity index 94%
copy from meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.12.bbappend
copy to meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.14.bbappend
index 2a6f090..b178b5e 100644
--- a/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.12.bbappend
+++ b/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.14.bbappend
@@ -6,6 +6,8 @@ require copy-defconfig.inc
 
 KERNEL_LOCALVERSION = "-g${@d.getVar('SRCPV', True).partition('+')[2][0:7]}"
 
+SRCREV = "${AUTOREV}"
+
 kernel_do_configure_prepend() {
     if [ ! -e ${B}/.scmversion -a ! -e ${S}/.scmversion ]
     then
-- 
1.7.9.5

Re: [PATCH 3/4] arago: unset ASSUME_PROVIDED for git-native

[Denys Dmytriyenko]

On Tue, Jul 15, 2014 at 06:28:34AM -0500, Chase Maupin wrote:
> * To get maximum size reduction use the later version of git
>   that is provided by git-native.  This reduces the kernel sources
>   size from 1002MB to 762MB.
> * Older versions of git work but they provide a depth of 1 for
>   all branches and tags instead of just for the branch requested.
> * This was tested with git version 1.7.9.5 and 1.9 as comparison
>   points.
> 
> Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> ---
>  meta-arago-distro/conf/distro/arago.conf |    5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/meta-arago-distro/conf/distro/arago.conf b/meta-arago-distro/conf/distro/arago.conf
> index b9af13f..1470f22 100644
> --- a/meta-arago-distro/conf/distro/arago.conf
> +++ b/meta-arago-distro/conf/distro/arago.conf
> @@ -50,6 +50,11 @@ require conf/distro/include/arago-prefs.inc
>  # Hack out sgx flag for now, as breaking on kernels >= 3.14
>  MACHINE_FEATURES := "${@filter_out('sgx','${MACHINE_FEATURES}',d)}"
>  
> +# Remove the ASSUME_PROVIDED setting for git-native so that we can
> +# use the latest version for the sourceipk which leads to a smaller
> +# git history.
> +ASSUME_PROVIDED := "${@filter_out('git-native','${ASSUME_PROVIDED}',d)}"

This now should work:

ASSUME_PROVIDED -= "git-native"


>  # Allow branding on top of Arago Distro and Core TI-SDK
>  # If ARAGO_BRAND is not set in local.conf, default to core
>  ARAGO_BRAND ??= "core"
> -- 
> 1.7.9.5
> 
> _______________________________________________
> meta-arago mailing list
> meta-arago@arago-project.org
> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 1/4] sourceipk: allow packaging shallow git clone

[Denys Dmytriyenko]

On Tue, Jul 15, 2014 at 06:28:32AM -0500, Chase Maupin wrote:
> * Add another flag SRCIPK_SHALLOW_CLONE that will do a shallow
>   clone of depth 1 to reduce the size of the git history.
>     * The full history can be fetched again using either of the
>       following commands based on the version of git being used
>         - git pull --unshallow (New git versions)
>         - git pull (older git versions)
> * Clean up some of the variable names to make it more clear what
>   they are storing.
> * Create a generic function to determine the fetch reporitory
> 
> Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> ---
>  meta-arago-distro/classes/sourceipk.bbclass |  100 ++++++++++++++++++++++++---
>  1 file changed, 89 insertions(+), 11 deletions(-)
> 
> diff --git a/meta-arago-distro/classes/sourceipk.bbclass b/meta-arago-distro/classes/sourceipk.bbclass
> index 20dda76..33878ac 100644
> --- a/meta-arago-distro/classes/sourceipk.bbclass
> +++ b/meta-arago-distro/classes/sourceipk.bbclass
> @@ -61,32 +61,111 @@ SRCIPK_INCLUDE_EXTRAFILES ?= "1"
>  
>  SRCIPK_PRESERVE_GIT ?= "false"
>  
> -adjust_git() {
> +# Create a shallow clone of the git repository to reduce the size of
> +# the sourceipk
> +SRCIPK_SHALLOW_CLONE ?= "false"
> +
> +# This function will return the fetch URL for a git repository passed as
> +# the first parameter.
> +get_remote() {
> +    git_repo="$1"
> +
> +    if [ "$git_repo" == "" ]
> +    then
> +        echo "git_repo not passed to get_remote"
> +        exit 1
> +    fi
> +
> +    cd $git_repo
> +
> +    # Get the remote repository fetch URL
> +    remote=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
> +
> +    # Since the echo'ed value of this statment is the returned value redirect
> +    # the output of this command to /dev/null
> +    cd - > /dev/null
> +
> +    # echo back the remote repository URL as the output of this function
> +    echo $remote
> +
> +    return 0
> +}
> +
> +# Some git repositories are very large and we do not want to ship the
> +# full history.  Instead we want to limit history to reduce the size while
> +# still keeping the git repository in place.  The full history can be
> +# fetched using git pull --unshallow or just git pull
> +# NOTE: This function depends on a git version >= 1.7.10.  It will work
> +#       with older versions but the size will be larger because rather
> +#       than just a single branch the limited history will be a depth of
> +#       1 for all branches and tags.
> +limit_git_history() {
> +    # By default limit the history to 1 commit since the user can always
> +    # use git pull --unshallow to fetch the rest of history.  The depth
> +    # level of 1 is set to keep from tracking through all merges and
> +    # pulling excess history
> +    commits="1"
> +
> +    # Temporary directory to make shallow clones in
> +    gitshallowclone="${WORKDIR}/temp-git-shallow-clone"
> +
> +    # Change directory to the git repository to be safe
> +    cd $tmp_dir/${SRCIPK_INSTALL_DIR}
> +
> +    # Create a temporary directory to hold the shallow git clone
> +    mkdir -p $gitshallowclone
>  
> +    remote=`get_remote $PWD`
> +
> +    git clone --depth $commits --branch ${BRANCH} file://$remote $gitshallowclone
> +
> +    # remove original kernel clone since we will replace it with the shallow
> +    # clone
> +    rm -rf $tmp_dir/${SRCIPK_INSTALL_DIR}/.git
> +
> +    # replace the original kernel git data with the shallow clone git data
> +    mv $gitshallowclone/.git $tmp_dir/${SRCIPK_INSTALL_DIR}/
> +    rm -rf $gitshallowclone
> +
> +    cd -
> +}
> +
> +adjust_git() {
>      orig_dir="$PWD"
>  
>      cd $tmp_dir/${SRCIPK_INSTALL_DIR}
>  
>      if [ -d ".git" ]
>      then
> +        # Get the location of the local repository
> +        local_repo=`get_remote $PWD`
>  
> -        # Grab path to cloned local repository
> -        old=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
> -
> -        if [ -d $old -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
> +        if [ -d $local_repo -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
>          then
> -            cd $old
> +            cd $local_repo
> +
> +            # If SRCIPK_SHALLOW_CLONE is true then make a shallow copy of the
> +            # git repository and then fix up the git URLs
> +            if [ "${SRCIPK_SHALLOW_CLONE}" == "true" ]
> +            then
> +                limit_git_history
> +            fi
>  
>              # Grab actual url used to create the repository
> -            orig=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
> +            remote_repo=`get_remote $PWD`
>  
>              cd -
>  
> -            git remote set-url origin $orig $old
> +            git remote set-url origin $remote_repo $local_repo
>  
> -            # Repackage the repository so its a proper clone of the original (remote) git repository
> +            # Repackage the repository so its a proper clone of the original
> +            # (remote) git repository
>              git repack -a -d



> -            rm .git/objects/info/alternates
> +
> +            if [ -e .git/objects/info/alternates ]
> +            then
> +                rm .git/objects/info/alternates
> +            fi

If you are worried about the file not being present:

rm -f .git/objects/info/alternates

Otherwise looks reasonable.


>  
>          else
>              rm -rf .git
> @@ -95,7 +174,6 @@ adjust_git() {
>      fi
>  
>      cd $orig_dir
> -
>  }
>  
>  # Create a README file that describes the contents of the source ipk
> -- 
> 1.7.9.5
> 
> _______________________________________________
> meta-arago mailing list
> meta-arago@arago-project.org
> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 1/4] sourceipk: allow packaging shallow git clone

[Cooper Jr., Franklin]

What branch/commit is checked out by default when a user uses this shallow cloned git repository?
Also since your manually cloning this repository what happens to patches that OE applies to the git repository or files that are manually copied?

I'm not sure if it is possible but it is probably safer to reuse the git repository as is from sourceipk and see if we can make that git repository shallow so we can preserve any changes to the repo.

> -----Original Message-----
> From: meta-arago-bounces@arago-project.org [mailto:meta-arago-
> bounces@arago-project.org] On Behalf Of Maupin, Chase
> Sent: Tuesday, July 15, 2014 6:29 AM
> To: meta-arago@arago-project.org
> Subject: [meta-arago] [PATCH 1/4] sourceipk: allow packaging shallow git
> clone
> 
> * Add another flag SRCIPK_SHALLOW_CLONE that will do a shallow
>   clone of depth 1 to reduce the size of the git history.
>     * The full history can be fetched again using either of the
>       following commands based on the version of git being used
>         - git pull --unshallow (New git versions)
>         - git pull (older git versions)
> * Clean up some of the variable names to make it more clear what
>   they are storing.
> * Create a generic function to determine the fetch reporitory
> 
> Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> ---
>  meta-arago-distro/classes/sourceipk.bbclass |  100
> ++++++++++++++++++++++++---
>  1 file changed, 89 insertions(+), 11 deletions(-)
> 
> diff --git a/meta-arago-distro/classes/sourceipk.bbclass b/meta-arago-
> distro/classes/sourceipk.bbclass
> index 20dda76..33878ac 100644
> --- a/meta-arago-distro/classes/sourceipk.bbclass
> +++ b/meta-arago-distro/classes/sourceipk.bbclass
> @@ -61,32 +61,111 @@ SRCIPK_INCLUDE_EXTRAFILES ?= "1"
> 
>  SRCIPK_PRESERVE_GIT ?= "false"
> 
> -adjust_git() {
> +# Create a shallow clone of the git repository to reduce the size of #
> +the sourceipk SRCIPK_SHALLOW_CLONE ?= "false"
> +
> +# This function will return the fetch URL for a git repository passed
> +as # the first parameter.
> +get_remote() {
> +    git_repo="$1"
> +
> +    if [ "$git_repo" == "" ]
> +    then
> +        echo "git_repo not passed to get_remote"
> +        exit 1
> +    fi
> +
> +    cd $git_repo
> +
> +    # Get the remote repository fetch URL
> +    remote=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
> +
> +    # Since the echo'ed value of this statment is the returned value redirect
> +    # the output of this command to /dev/null
> +    cd - > /dev/null
> +
> +    # echo back the remote repository URL as the output of this function
> +    echo $remote
> +
> +    return 0
> +}
> +
> +# Some git repositories are very large and we do not want to ship the #
> +full history.  Instead we want to limit history to reduce the size
> +while # still keeping the git repository in place.  The full history
> +can be # fetched using git pull --unshallow or just git pull # NOTE:
> +This function depends on a git version >= 1.7.10.  It will work
> +#       with older versions but the size will be larger because rather
> +#       than just a single branch the limited history will be a depth of
> +#       1 for all branches and tags.
> +limit_git_history() {
> +    # By default limit the history to 1 commit since the user can always
> +    # use git pull --unshallow to fetch the rest of history.  The depth
> +    # level of 1 is set to keep from tracking through all merges and
> +    # pulling excess history
> +    commits="1"
> +
> +    # Temporary directory to make shallow clones in
> +    gitshallowclone="${WORKDIR}/temp-git-shallow-clone"
> +
> +    # Change directory to the git repository to be safe
> +    cd $tmp_dir/${SRCIPK_INSTALL_DIR}
> +
> +    # Create a temporary directory to hold the shallow git clone
> +    mkdir -p $gitshallowclone
> 
> +    remote=`get_remote $PWD`
> +
> +    git clone --depth $commits --branch ${BRANCH} file://$remote
> + $gitshallowclone
> +
> +    # remove original kernel clone since we will replace it with the shallow
> +    # clone
> +    rm -rf $tmp_dir/${SRCIPK_INSTALL_DIR}/.git
> +
> +    # replace the original kernel git data with the shallow clone git data
> +    mv $gitshallowclone/.git $tmp_dir/${SRCIPK_INSTALL_DIR}/
> +    rm -rf $gitshallowclone
> +
> +    cd -
> +}
> +
> +adjust_git() {
>      orig_dir="$PWD"
> 
>      cd $tmp_dir/${SRCIPK_INSTALL_DIR}
> 
>      if [ -d ".git" ]
>      then
> +        # Get the location of the local repository
> +        local_repo=`get_remote $PWD`
> 
> -        # Grab path to cloned local repository
> -        old=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
> -
> -        if [ -d $old -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
> +        if [ -d $local_repo -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
>          then
> -            cd $old
> +            cd $local_repo
> +
> +            # If SRCIPK_SHALLOW_CLONE is true then make a shallow copy of the
> +            # git repository and then fix up the git URLs
> +            if [ "${SRCIPK_SHALLOW_CLONE}" == "true" ]
> +            then
> +                limit_git_history
> +            fi
> 
>              # Grab actual url used to create the repository
> -            orig=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
> +            remote_repo=`get_remote $PWD`
> 
>              cd -
> 
> -            git remote set-url origin $orig $old
> +            git remote set-url origin $remote_repo $local_repo
> 
> -            # Repackage the repository so its a proper clone of the original
> (remote) git repository
> +            # Repackage the repository so its a proper clone of the original
> +            # (remote) git repository
>              git repack -a -d
> -            rm .git/objects/info/alternates
> +
> +            if [ -e .git/objects/info/alternates ]
> +            then
> +                rm .git/objects/info/alternates
> +            fi
> 
>          else
>              rm -rf .git
> @@ -95,7 +174,6 @@ adjust_git() {
>      fi
> 
>      cd $orig_dir
> -
>  }
> 
>  # Create a README file that describes the contents of the source ipk
> --
> 1.7.9.5
> 
> _______________________________________________
> meta-arago mailing list
> meta-arago@arago-project.org
> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 4/4] linux-ti-staging: update bbappend for 3.14 version

[Cooper Jr., Franklin]

> -----Original Message-----
> From: meta-arago-bounces@arago-project.org [mailto:meta-arago-
> bounces@arago-project.org] On Behalf Of Maupin, Chase
> Sent: Tuesday, July 15, 2014 6:29 AM
> To: meta-arago@arago-project.org
> Subject: [meta-arago] [PATCH 4/4] linux-ti-staging: update bbappend for 3.14
> version
> 
> * Update the bbappend for the linux-ti-staging to work with the
>   3.14 kernel version.
> * Set the kernel to AUTOREV while developing the latest 3.14 kernel
> 
> Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> ---
>  ....12.bbappend => linux-ti-staging_3.14.bbappend} |    2 ++
>  1 file changed, 2 insertions(+)
>  copy meta-arago-distro/recipes-kernel/linux/{linux-ti-staging_3.12.bbappend
> => linux-ti-staging_3.14.bbappend} (94%)
> 
> diff --git a/meta-arago-distro/recipes-kernel/linux/linux-ti-
> staging_3.12.bbappend b/meta-arago-distro/recipes-kernel/linux/linux-ti-
> staging_3.14.bbappend
> similarity index 94%
> copy from meta-arago-distro/recipes-kernel/linux/linux-ti-
> staging_3.12.bbappend
> copy to meta-arago-distro/recipes-kernel/linux/linux-ti-
> staging_3.14.bbappend
> index 2a6f090..b178b5e 100644
> --- a/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.12.bbappend
> +++ b/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.14.bbappend
> @@ -6,6 +6,8 @@ require copy-defconfig.inc
> 
>  KERNEL_LOCALVERSION = "-g${@d.getVar('SRCPV', True).partition('+')[2][0:7]}"
> 
[Franklin] PR bump missing.
> +SRCREV = "${AUTOREV}"
> +
>  kernel_do_configure_prepend() {
>      if [ ! -e ${B}/.scmversion -a ! -e ${S}/.scmversion ]
>      then
> --
> 1.7.9.5
> 
> _______________________________________________
> meta-arago mailing list
> meta-arago@arago-project.org
> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 4/4] linux-ti-staging: update bbappend for 3.14 version

[Denys Dmytriyenko]

On Wed, Jul 16, 2014 at 02:30:56PM +0000, Cooper Jr., Franklin wrote:
> 
> 
> > -----Original Message-----
> > From: meta-arago-bounces@arago-project.org [mailto:meta-arago-
> > bounces@arago-project.org] On Behalf Of Maupin, Chase
> > Sent: Tuesday, July 15, 2014 6:29 AM
> > To: meta-arago@arago-project.org
> > Subject: [meta-arago] [PATCH 4/4] linux-ti-staging: update bbappend for 3.14
> > version
> > 
> > * Update the bbappend for the linux-ti-staging to work with the
> >   3.14 kernel version.
> > * Set the kernel to AUTOREV while developing the latest 3.14 kernel
> > 
> > Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> > ---
> >  ....12.bbappend => linux-ti-staging_3.14.bbappend} |    2 ++
> >  1 file changed, 2 insertions(+)
> >  copy meta-arago-distro/recipes-kernel/linux/{linux-ti-staging_3.12.bbappend
> > => linux-ti-staging_3.14.bbappend} (94%)
> > 
> > diff --git a/meta-arago-distro/recipes-kernel/linux/linux-ti-
> > staging_3.12.bbappend b/meta-arago-distro/recipes-kernel/linux/linux-ti-
> > staging_3.14.bbappend
> > similarity index 94%
> > copy from meta-arago-distro/recipes-kernel/linux/linux-ti-
> > staging_3.12.bbappend
> > copy to meta-arago-distro/recipes-kernel/linux/linux-ti-
> > staging_3.14.bbappend
> > index 2a6f090..b178b5e 100644
> > --- a/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.12.bbappend
> > +++ b/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.14.bbappend
> > @@ -6,6 +6,8 @@ require copy-defconfig.inc
> > 
> >  KERNEL_LOCALVERSION = "-g${@d.getVar('SRCPV', True).partition('+')[2][0:7]}"
> > 
> [Franklin] PR bump missing.

How so?


> > +SRCREV = "${AUTOREV}"
> > +
> >  kernel_do_configure_prepend() {
> >      if [ ! -e ${B}/.scmversion -a ! -e ${S}/.scmversion ]
> >      then
> > --
> > 1.7.9.5
> > 
> > _______________________________________________
> > meta-arago mailing list
> > meta-arago@arago-project.org
> > http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago
> _______________________________________________
> meta-arago mailing list
> meta-arago@arago-project.org
> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 4/4] linux-ti-staging: update bbappend for 3.14 version

[Cooper Jr., Franklin]

> -----Original Message-----
> From: Dmytriyenko, Denys
> Sent: Wednesday, July 16, 2014 10:02 AM
> To: Cooper Jr., Franklin
> Cc: Maupin, Chase; meta-arago@arago-project.org
> Subject: Re: [meta-arago] [PATCH 4/4] linux-ti-staging: update bbappend for
> 3.14 version
> 
> On Wed, Jul 16, 2014 at 02:30:56PM +0000, Cooper Jr., Franklin wrote:
> >
> >
> > > -----Original Message-----
> > > From: meta-arago-bounces@arago-project.org [mailto:meta-arago-
> > > bounces@arago-project.org] On Behalf Of Maupin, Chase
> > > Sent: Tuesday, July 15, 2014 6:29 AM
> > > To: meta-arago@arago-project.org
> > > Subject: [meta-arago] [PATCH 4/4] linux-ti-staging: update bbappend
> > > for 3.14 version
> > >
> > > * Update the bbappend for the linux-ti-staging to work with the
> > >   3.14 kernel version.
> > > * Set the kernel to AUTOREV while developing the latest 3.14 kernel
> > >
> > > Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> > > ---
> > >  ....12.bbappend => linux-ti-staging_3.14.bbappend} |    2 ++
> > >  1 file changed, 2 insertions(+)
> > >  copy
> > > meta-arago-distro/recipes-kernel/linux/{linux-ti-staging_3.12.bbappe
> > > nd => linux-ti-staging_3.14.bbappend} (94%)
> > >
> > > diff --git a/meta-arago-distro/recipes-kernel/linux/linux-ti-
> > > staging_3.12.bbappend
> > > b/meta-arago-distro/recipes-kernel/linux/linux-ti-
> > > staging_3.14.bbappend
> > > similarity index 94%
> > > copy from meta-arago-distro/recipes-kernel/linux/linux-ti-
> > > staging_3.12.bbappend
> > > copy to meta-arago-distro/recipes-kernel/linux/linux-ti-
> > > staging_3.14.bbappend
> > > index 2a6f090..b178b5e 100644
> > > ---
> > > a/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.12.bbapp
> > > end
> > > +++ b/meta-arago-distro/recipes-kernel/linux/linux-ti-staging_3.14.b
> > > +++ bappend
> > > @@ -6,6 +6,8 @@ require copy-defconfig.inc
> > >
> > >  KERNEL_LOCALVERSION = "-g${@d.getVar('SRCPV',
> True).partition('+')[2][0:7]}"
> > >
> > [Franklin] PR bump missing.
> 
> How so?
[Franklin] Sorry missed the fact that this is renaming the recipe. Its fine as is.
> 
> 
> > > +SRCREV = "${AUTOREV}"
> > > +
> > >  kernel_do_configure_prepend() {
> > >      if [ ! -e ${B}/.scmversion -a ! -e ${S}/.scmversion ]
> > >      then
> > > --
> > > 1.7.9.5
> > >
> > > _______________________________________________
> > > meta-arago mailing list
> > > meta-arago@arago-project.org
> > > http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago
> > _______________________________________________
> > meta-arago mailing list
> > meta-arago@arago-project.org
> > http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 3/4] arago: unset ASSUME_PROVIDED for git-native

[Denys Dmytriyenko]

On Tue, Jul 15, 2014 at 01:28:22PM -0400, Denys Dmytriyenko wrote:
> On Tue, Jul 15, 2014 at 06:28:34AM -0500, Chase Maupin wrote:
> > * To get maximum size reduction use the later version of git
> >   that is provided by git-native.  This reduces the kernel sources
> >   size from 1002MB to 762MB.
> > * Older versions of git work but they provide a depth of 1 for
> >   all branches and tags instead of just for the branch requested.
> > * This was tested with git version 1.7.9.5 and 1.9 as comparison
> >   points.
> > 
> > Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> > ---
> >  meta-arago-distro/conf/distro/arago.conf |    5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/meta-arago-distro/conf/distro/arago.conf b/meta-arago-distro/conf/distro/arago.conf
> > index b9af13f..1470f22 100644
> > --- a/meta-arago-distro/conf/distro/arago.conf
> > +++ b/meta-arago-distro/conf/distro/arago.conf
> > @@ -50,6 +50,11 @@ require conf/distro/include/arago-prefs.inc
> >  # Hack out sgx flag for now, as breaking on kernels >= 3.14
> >  MACHINE_FEATURES := "${@filter_out('sgx','${MACHINE_FEATURES}',d)}"
> >  
> > +# Remove the ASSUME_PROVIDED setting for git-native so that we can
> > +# use the latest version for the sourceipk which leads to a smaller
> > +# git history.
> > +ASSUME_PROVIDED := "${@filter_out('git-native','${ASSUME_PROVIDED}',d)}"
> 
> This now should work:
> 
> ASSUME_PROVIDED -= "git-native"

Correction:

ASSUME_PROVIDED_remove = "git-native"


> >  # Allow branding on top of Arago Distro and Core TI-SDK
> >  # If ARAGO_BRAND is not set in local.conf, default to core
> >  ARAGO_BRAND ??= "core"
> > -- 
> > 1.7.9.5
> > 
> > _______________________________________________
> > meta-arago mailing list
> > meta-arago@arago-project.org
> > http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago
> _______________________________________________
> meta-arago mailing list
> meta-arago@arago-project.org
> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 1/4] sourceipk: allow packaging shallow git clone

[Maupin, Chase]

I will reply to this in detail later. I'm traveling and in meetings. But yes, extra files are preserved. 
> On Jul 16, 2014, at 9:19 AM, "Cooper Jr., Franklin" <fcooper@ti.com> wrote:
> 
> What branch/commit is checked out by default when a user uses this shallow cloned git repository?
> Also since your manually cloning this repository what happens to patches that OE applies to the git repository or files that are manually copied?
> 
> I'm not sure if it is possible but it is probably safer to reuse the git repository as is from sourceipk and see if we can make that git repository shallow so we can preserve any changes to the repo.
> 
>> -----Original Message-----
>> From: meta-arago-bounces@arago-project.org [mailto:meta-arago-
>> bounces@arago-project.org] On Behalf Of Maupin, Chase
>> Sent: Tuesday, July 15, 2014 6:29 AM
>> To: meta-arago@arago-project.org
>> Subject: [meta-arago] [PATCH 1/4] sourceipk: allow packaging shallow git
>> clone
>> 
>> * Add another flag SRCIPK_SHALLOW_CLONE that will do a shallow
>>  clone of depth 1 to reduce the size of the git history.
>>    * The full history can be fetched again using either of the
>>      following commands based on the version of git being used
>>        - git pull --unshallow (New git versions)
>>        - git pull (older git versions)
>> * Clean up some of the variable names to make it more clear what
>>  they are storing.
>> * Create a generic function to determine the fetch reporitory
>> 
>> Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
>> ---
>> meta-arago-distro/classes/sourceipk.bbclass |  100
>> ++++++++++++++++++++++++---
>> 1 file changed, 89 insertions(+), 11 deletions(-)
>> 
>> diff --git a/meta-arago-distro/classes/sourceipk.bbclass b/meta-arago-
>> distro/classes/sourceipk.bbclass
>> index 20dda76..33878ac 100644
>> --- a/meta-arago-distro/classes/sourceipk.bbclass
>> +++ b/meta-arago-distro/classes/sourceipk.bbclass
>> @@ -61,32 +61,111 @@ SRCIPK_INCLUDE_EXTRAFILES ?= "1"
>> 
>> SRCIPK_PRESERVE_GIT ?= "false"
>> 
>> -adjust_git() {
>> +# Create a shallow clone of the git repository to reduce the size of #
>> +the sourceipk SRCIPK_SHALLOW_CLONE ?= "false"
>> +
>> +# This function will return the fetch URL for a git repository passed
>> +as # the first parameter.
>> +get_remote() {
>> +    git_repo="$1"
>> +
>> +    if [ "$git_repo" == "" ]
>> +    then
>> +        echo "git_repo not passed to get_remote"
>> +        exit 1
>> +    fi
>> +
>> +    cd $git_repo
>> +
>> +    # Get the remote repository fetch URL
>> +    remote=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
>> +
>> +    # Since the echo'ed value of this statment is the returned value redirect
>> +    # the output of this command to /dev/null
>> +    cd - > /dev/null
>> +
>> +    # echo back the remote repository URL as the output of this function
>> +    echo $remote
>> +
>> +    return 0
>> +}
>> +
>> +# Some git repositories are very large and we do not want to ship the #
>> +full history.  Instead we want to limit history to reduce the size
>> +while # still keeping the git repository in place.  The full history
>> +can be # fetched using git pull --unshallow or just git pull # NOTE:
>> +This function depends on a git version >= 1.7.10.  It will work
>> +#       with older versions but the size will be larger because rather
>> +#       than just a single branch the limited history will be a depth of
>> +#       1 for all branches and tags.
>> +limit_git_history() {
>> +    # By default limit the history to 1 commit since the user can always
>> +    # use git pull --unshallow to fetch the rest of history.  The depth
>> +    # level of 1 is set to keep from tracking through all merges and
>> +    # pulling excess history
>> +    commits="1"
>> +
>> +    # Temporary directory to make shallow clones in
>> +    gitshallowclone="${WORKDIR}/temp-git-shallow-clone"
>> +
>> +    # Change directory to the git repository to be safe
>> +    cd $tmp_dir/${SRCIPK_INSTALL_DIR}
>> +
>> +    # Create a temporary directory to hold the shallow git clone
>> +    mkdir -p $gitshallowclone
>> 
>> +    remote=`get_remote $PWD`
>> +
>> +    git clone --depth $commits --branch ${BRANCH} file://$remote
>> + $gitshallowclone
>> +
>> +    # remove original kernel clone since we will replace it with the shallow
>> +    # clone
>> +    rm -rf $tmp_dir/${SRCIPK_INSTALL_DIR}/.git
>> +
>> +    # replace the original kernel git data with the shallow clone git data
>> +    mv $gitshallowclone/.git $tmp_dir/${SRCIPK_INSTALL_DIR}/
>> +    rm -rf $gitshallowclone
>> +
>> +    cd -
>> +}
>> +
>> +adjust_git() {
>>     orig_dir="$PWD"
>> 
>>     cd $tmp_dir/${SRCIPK_INSTALL_DIR}
>> 
>>     if [ -d ".git" ]
>>     then
>> +        # Get the location of the local repository
>> +        local_repo=`get_remote $PWD`
>> 
>> -        # Grab path to cloned local repository
>> -        old=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
>> -
>> -        if [ -d $old -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
>> +        if [ -d $local_repo -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
>>         then
>> -            cd $old
>> +            cd $local_repo
>> +
>> +            # If SRCIPK_SHALLOW_CLONE is true then make a shallow copy of the
>> +            # git repository and then fix up the git URLs
>> +            if [ "${SRCIPK_SHALLOW_CLONE}" == "true" ]
>> +            then
>> +                limit_git_history
>> +            fi
>> 
>>             # Grab actual url used to create the repository
>> -            orig=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1   | cut -c 7- | tr -d ' '`
>> +            remote_repo=`get_remote $PWD`
>> 
>>             cd -
>> 
>> -            git remote set-url origin $orig $old
>> +            git remote set-url origin $remote_repo $local_repo
>> 
>> -            # Repackage the repository so its a proper clone of the original
>> (remote) git repository
>> +            # Repackage the repository so its a proper clone of the original
>> +            # (remote) git repository
>>             git repack -a -d
>> -            rm .git/objects/info/alternates
>> +
>> +            if [ -e .git/objects/info/alternates ]
>> +            then
>> +                rm .git/objects/info/alternates
>> +            fi
>> 
>>         else
>>             rm -rf .git
>> @@ -95,7 +174,6 @@ adjust_git() {
>>     fi
>> 
>>     cd $orig_dir
>> -
>> }
>> 
>> # Create a README file that describes the contents of the source ipk
>> --
>> 1.7.9.5
>> 
>> _______________________________________________
>> meta-arago mailing list
>> meta-arago@arago-project.org
>> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 1/4] sourceipk: allow packaging shallow git clone

[Maupin, Chase]

>-----Original Message-----
>From: Cooper Jr., Franklin
>Sent: Wednesday, July 16, 2014 9:19 AM
>To: Maupin, Chase; meta-arago@arago-project.org
>Subject: RE: [meta-arago] [PATCH 1/4] sourceipk: allow packaging
>shallow git clone
>
>What branch/commit is checked out by default when a user uses this
>shallow cloned git repository?
>Also since your manually cloning this repository what happens to
>patches that OE applies to the git repository or files that are
>manually copied?

Default is that the ${BRANCH} is checked out at the ${SRCREV}.

The copied files like the config are still maintained because we replace the git data in the original sourceipk directory rather than doing a whole new copy.  So if the files were there before they are still there vs. doing a new checkout where they would be lost.

>
>I'm not sure if it is possible but it is probably safer to reuse
>the git repository as is from sourceipk and see if we can make
>that git repository shallow so we can preserve any changes to the
>repo.

I had not found a way to take an existing repo and turn it into a shallow clone.  That is why I make the shallow clone separate and then replace the git repo with the shallow clone data.

>
>> -----Original Message-----
>> From: meta-arago-bounces@arago-project.org [mailto:meta-arago-
>> bounces@arago-project.org] On Behalf Of Maupin, Chase
>> Sent: Tuesday, July 15, 2014 6:29 AM
>> To: meta-arago@arago-project.org
>> Subject: [meta-arago] [PATCH 1/4] sourceipk: allow packaging
>shallow git
>> clone
>>
>> * Add another flag SRCIPK_SHALLOW_CLONE that will do a shallow
>>   clone of depth 1 to reduce the size of the git history.
>>     * The full history can be fetched again using either of the
>>       following commands based on the version of git being used
>>         - git pull --unshallow (New git versions)
>>         - git pull (older git versions)
>> * Clean up some of the variable names to make it more clear what
>>   they are storing.
>> * Create a generic function to determine the fetch reporitory
>>
>> Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
>> ---
>>  meta-arago-distro/classes/sourceipk.bbclass |  100
>> ++++++++++++++++++++++++---
>>  1 file changed, 89 insertions(+), 11 deletions(-)
>>
>> diff --git a/meta-arago-distro/classes/sourceipk.bbclass b/meta-
>arago-
>> distro/classes/sourceipk.bbclass
>> index 20dda76..33878ac 100644
>> --- a/meta-arago-distro/classes/sourceipk.bbclass
>> +++ b/meta-arago-distro/classes/sourceipk.bbclass
>> @@ -61,32 +61,111 @@ SRCIPK_INCLUDE_EXTRAFILES ?= "1"
>>
>>  SRCIPK_PRESERVE_GIT ?= "false"
>>
>> -adjust_git() {
>> +# Create a shallow clone of the git repository to reduce the
>size of #
>> +the sourceipk SRCIPK_SHALLOW_CLONE ?= "false"
>> +
>> +# This function will return the fetch URL for a git repository
>passed
>> +as # the first parameter.
>> +get_remote() {
>> +    git_repo="$1"
>> +
>> +    if [ "$git_repo" == "" ]
>> +    then
>> +        echo "git_repo not passed to get_remote"
>> +        exit 1
>> +    fi
>> +
>> +    cd $git_repo
>> +
>> +    # Get the remote repository fetch URL
>> +    remote=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1
>| cut -c 7- | tr -d ' '`
>> +
>> +    # Since the echo'ed value of this statment is the returned
>value redirect
>> +    # the output of this command to /dev/null
>> +    cd - > /dev/null
>> +
>> +    # echo back the remote repository URL as the output of this
>function
>> +    echo $remote
>> +
>> +    return 0
>> +}
>> +
>> +# Some git repositories are very large and we do not want to
>ship the #
>> +full history.  Instead we want to limit history to reduce the
>size
>> +while # still keeping the git repository in place.  The full
>history
>> +can be # fetched using git pull --unshallow or just git pull #
>NOTE:
>> +This function depends on a git version >= 1.7.10.  It will work
>> +#       with older versions but the size will be larger because
>rather
>> +#       than just a single branch the limited history will be a
>depth of
>> +#       1 for all branches and tags.
>> +limit_git_history() {
>> +    # By default limit the history to 1 commit since the user
>can always
>> +    # use git pull --unshallow to fetch the rest of history.
>The depth
>> +    # level of 1 is set to keep from tracking through all
>merges and
>> +    # pulling excess history
>> +    commits="1"
>> +
>> +    # Temporary directory to make shallow clones in
>> +    gitshallowclone="${WORKDIR}/temp-git-shallow-clone"
>> +
>> +    # Change directory to the git repository to be safe
>> +    cd $tmp_dir/${SRCIPK_INSTALL_DIR}
>> +
>> +    # Create a temporary directory to hold the shallow git
>clone
>> +    mkdir -p $gitshallowclone
>>
>> +    remote=`get_remote $PWD`
>> +
>> +    git clone --depth $commits --branch ${BRANCH}
>file://$remote
>> + $gitshallowclone
>> +
>> +    # remove original kernel clone since we will replace it
>with the shallow
>> +    # clone
>> +    rm -rf $tmp_dir/${SRCIPK_INSTALL_DIR}/.git
>> +
>> +    # replace the original kernel git data with the shallow
>clone git data
>> +    mv $gitshallowclone/.git $tmp_dir/${SRCIPK_INSTALL_DIR}/
>> +    rm -rf $gitshallowclone
>> +
>> +    cd -
>> +}
>> +
>> +adjust_git() {
>>      orig_dir="$PWD"
>>
>>      cd $tmp_dir/${SRCIPK_INSTALL_DIR}
>>
>>      if [ -d ".git" ]
>>      then
>> +        # Get the location of the local repository
>> +        local_repo=`get_remote $PWD`
>>
>> -        # Grab path to cloned local repository
>> -        old=`git remote -v | grep "(fetch)" | cut -d ' ' -f 1
>| cut -c 7- | tr -d ' '`
>> -
>> -        if [ -d $old -a "${SRCIPK_PRESERVE_GIT}" = "true" ]
>> +        if [ -d $local_repo -a "${SRCIPK_PRESERVE_GIT}" =
>"true" ]
>>          then
>> -            cd $old
>> +            cd $local_repo
>> +
>> +            # If SRCIPK_SHALLOW_CLONE is true then make a
>shallow copy of the
>> +            # git repository and then fix up the git URLs
>> +            if [ "${SRCIPK_SHALLOW_CLONE}" == "true" ]
>> +            then
>> +                limit_git_history
>> +            fi
>>
>>              # Grab actual url used to create the repository
>> -            orig=`git remote -v | grep "(fetch)" | cut -d ' ' -
>f 1   | cut -c 7- | tr -d ' '`
>> +            remote_repo=`get_remote $PWD`
>>
>>              cd -
>>
>> -            git remote set-url origin $orig $old
>> +            git remote set-url origin $remote_repo $local_repo
>>
>> -            # Repackage the repository so its a proper clone of
>the original
>> (remote) git repository
>> +            # Repackage the repository so its a proper clone of
>the original
>> +            # (remote) git repository
>>              git repack -a -d
>> -            rm .git/objects/info/alternates
>> +
>> +            if [ -e .git/objects/info/alternates ]
>> +            then
>> +                rm .git/objects/info/alternates
>> +            fi
>>
>>          else
>>              rm -rf .git
>> @@ -95,7 +174,6 @@ adjust_git() {
>>      fi
>>
>>      cd $orig_dir
>> -
>>  }
>>
>>  # Create a README file that describes the contents of the
>source ipk
>> --
>> 1.7.9.5
>>
>> _______________________________________________
>> meta-arago mailing list
>> meta-arago@arago-project.org
>> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 3/4] arago: unset ASSUME_PROVIDED for git-native

[Maupin, Chase]

>-----Original Message-----
>From: Dmytriyenko, Denys
>Sent: Thursday, July 17, 2014 12:06 PM
>To: Maupin, Chase
>Cc: meta-arago@arago-project.org
>Subject: Re: [meta-arago] [PATCH 3/4] arago: unset ASSUME_PROVIDED
>for git-native
>
>On Tue, Jul 15, 2014 at 01:28:22PM -0400, Denys Dmytriyenko wrote:
>> On Tue, Jul 15, 2014 at 06:28:34AM -0500, Chase Maupin wrote:
>> > * To get maximum size reduction use the later version of git
>> >   that is provided by git-native.  This reduces the kernel
>sources
>> >   size from 1002MB to 762MB.
>> > * Older versions of git work but they provide a depth of 1 for
>> >   all branches and tags instead of just for the branch
>requested.
>> > * This was tested with git version 1.7.9.5 and 1.9 as
>comparison
>> >   points.
>> >
>> > Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
>> > ---
>> >  meta-arago-distro/conf/distro/arago.conf |    5 +++++
>> >  1 file changed, 5 insertions(+)
>> >
>> > diff --git a/meta-arago-distro/conf/distro/arago.conf b/meta-
>arago-distro/conf/distro/arago.conf
>> > index b9af13f..1470f22 100644
>> > --- a/meta-arago-distro/conf/distro/arago.conf
>> > +++ b/meta-arago-distro/conf/distro/arago.conf
>> > @@ -50,6 +50,11 @@ require conf/distro/include/arago-prefs.inc
>> >  # Hack out sgx flag for now, as breaking on kernels >= 3.14
>> >  MACHINE_FEATURES :=
>"${@filter_out('sgx','${MACHINE_FEATURES}',d)}"
>> >
>> > +# Remove the ASSUME_PROVIDED setting for git-native so that
>we can
>> > +# use the latest version for the sourceipk which leads to a
>smaller
>> > +# git history.
>> > +ASSUME_PROVIDED := "${@filter_out('git-
>native','${ASSUME_PROVIDED}',d)}"
>>
>> This now should work:
>>
>> ASSUME_PROVIDED -= "git-native"
>
>Correction:
>
>ASSUME_PROVIDED_remove = "git-native"

This does not seem to have worked.  I don't see git-native being built when I use this.

>
>
>> >  # Allow branding on top of Arago Distro and Core TI-SDK
>> >  # If ARAGO_BRAND is not set in local.conf, default to core
>> >  ARAGO_BRAND ??= "core"
>> > --
>> > 1.7.9.5
>> >
>> > _______________________________________________
>> > meta-arago mailing list
>> > meta-arago@arago-project.org
>> > http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago
>> _______________________________________________
>> meta-arago mailing list
>> meta-arago@arago-project.org
>> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 3/4] arago: unset ASSUME_PROVIDED for git-native

[Denys Dmytriyenko]

On Fri, Jul 18, 2014 at 11:52:37AM -0400, Maupin, Chase wrote:
> >-----Original Message-----
> >From: Dmytriyenko, Denys
> >Sent: Thursday, July 17, 2014 12:06 PM
> >To: Maupin, Chase
> >Cc: meta-arago@arago-project.org
> >Subject: Re: [meta-arago] [PATCH 3/4] arago: unset ASSUME_PROVIDED
> >for git-native
> >
> >On Tue, Jul 15, 2014 at 01:28:22PM -0400, Denys Dmytriyenko wrote:
> >> On Tue, Jul 15, 2014 at 06:28:34AM -0500, Chase Maupin wrote:
> >> > * To get maximum size reduction use the later version of git
> >> >   that is provided by git-native.  This reduces the kernel
> >sources
> >> >   size from 1002MB to 762MB.
> >> > * Older versions of git work but they provide a depth of 1 for
> >> >   all branches and tags instead of just for the branch
> >requested.
> >> > * This was tested with git version 1.7.9.5 and 1.9 as
> >comparison
> >> >   points.
> >> >
> >> > Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
> >> > ---
> >> >  meta-arago-distro/conf/distro/arago.conf |    5 +++++
> >> >  1 file changed, 5 insertions(+)
> >> >
> >> > diff --git a/meta-arago-distro/conf/distro/arago.conf b/meta-
> >arago-distro/conf/distro/arago.conf
> >> > index b9af13f..1470f22 100644
> >> > --- a/meta-arago-distro/conf/distro/arago.conf
> >> > +++ b/meta-arago-distro/conf/distro/arago.conf
> >> > @@ -50,6 +50,11 @@ require conf/distro/include/arago-prefs.inc
> >> >  # Hack out sgx flag for now, as breaking on kernels >= 3.14
> >> >  MACHINE_FEATURES :=
> >"${@filter_out('sgx','${MACHINE_FEATURES}',d)}"
> >> >
> >> > +# Remove the ASSUME_PROVIDED setting for git-native so that
> >we can
> >> > +# use the latest version for the sourceipk which leads to a
> >smaller
> >> > +# git history.
> >> > +ASSUME_PROVIDED := "${@filter_out('git-
> >native','${ASSUME_PROVIDED}',d)}"
> >>
> >> This now should work:
> >>
> >> ASSUME_PROVIDED -= "git-native"
> >
> >Correction:
> >
> >ASSUME_PROVIDED_remove = "git-native"
> 
> This does not seem to have worked.  I don't see git-native being built when I use this.

Well, darn it, looks like bitbake processes ASSUME_PROVIDED list before 
resolving all the overrides... :(


> >> >  # Allow branding on top of Arago Distro and Core TI-SDK
> >> >  # If ARAGO_BRAND is not set in local.conf, default to core
> >> >  ARAGO_BRAND ??= "core"
> >> > --
> >> > 1.7.9.5
> >> >
> >> > _______________________________________________
> >> > meta-arago mailing list
> >> > meta-arago@arago-project.org
> >> > http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago
> >> _______________________________________________
> >> meta-arago mailing list
> >> meta-arago@arago-project.org
> >> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago

Re: [PATCH 3/4] arago: unset ASSUME_PROVIDED for git-native

[Maupin, Chase]

>-----Original Message-----
>From: Dmytriyenko, Denys
>Sent: Friday, July 18, 2014 12:55 PM
>To: Maupin, Chase
>Cc: meta-arago@arago-project.org
>Subject: Re: [meta-arago] [PATCH 3/4] arago: unset ASSUME_PROVIDED
>for git-native
>
>On Fri, Jul 18, 2014 at 11:52:37AM -0400, Maupin, Chase wrote:
>> >-----Original Message-----
>> >From: Dmytriyenko, Denys
>> >Sent: Thursday, July 17, 2014 12:06 PM
>> >To: Maupin, Chase
>> >Cc: meta-arago@arago-project.org
>> >Subject: Re: [meta-arago] [PATCH 3/4] arago: unset
>ASSUME_PROVIDED
>> >for git-native
>> >
>> >On Tue, Jul 15, 2014 at 01:28:22PM -0400, Denys Dmytriyenko
>wrote:
>> >> On Tue, Jul 15, 2014 at 06:28:34AM -0500, Chase Maupin wrote:
>> >> > * To get maximum size reduction use the later version of
>git
>> >> >   that is provided by git-native.  This reduces the kernel
>> >sources
>> >> >   size from 1002MB to 762MB.
>> >> > * Older versions of git work but they provide a depth of 1
>for
>> >> >   all branches and tags instead of just for the branch
>> >requested.
>> >> > * This was tested with git version 1.7.9.5 and 1.9 as
>> >comparison
>> >> >   points.
>> >> >
>> >> > Signed-off-by: Chase Maupin <Chase.Maupin@ti.com>
>> >> > ---
>> >> >  meta-arago-distro/conf/distro/arago.conf |    5 +++++
>> >> >  1 file changed, 5 insertions(+)
>> >> >
>> >> > diff --git a/meta-arago-distro/conf/distro/arago.conf
>b/meta-
>> >arago-distro/conf/distro/arago.conf
>> >> > index b9af13f..1470f22 100644
>> >> > --- a/meta-arago-distro/conf/distro/arago.conf
>> >> > +++ b/meta-arago-distro/conf/distro/arago.conf
>> >> > @@ -50,6 +50,11 @@ require conf/distro/include/arago-
>prefs.inc
>> >> >  # Hack out sgx flag for now, as breaking on kernels >=
>3.14
>> >> >  MACHINE_FEATURES :=
>> >"${@filter_out('sgx','${MACHINE_FEATURES}',d)}"
>> >> >
>> >> > +# Remove the ASSUME_PROVIDED setting for git-native so
>that
>> >we can
>> >> > +# use the latest version for the sourceipk which leads to
>a
>> >smaller
>> >> > +# git history.
>> >> > +ASSUME_PROVIDED := "${@filter_out('git-
>> >native','${ASSUME_PROVIDED}',d)}"
>> >>
>> >> This now should work:
>> >>
>> >> ASSUME_PROVIDED -= "git-native"
>> >
>> >Correction:
>> >
>> >ASSUME_PROVIDED_remove = "git-native"
>>
>> This does not seem to have worked.  I don't see git-native being
>built when I use this.
>
>Well, darn it, looks like bitbake processes ASSUME_PROVIDED list
>before
>resolving all the overrides... :(

I'll send an update of patch 1 and leave this one alone then.

>
>
>> >> >  # Allow branding on top of Arago Distro and Core TI-SDK
>> >> >  # If ARAGO_BRAND is not set in local.conf, default to core
>> >> >  ARAGO_BRAND ??= "core"
>> >> > --
>> >> > 1.7.9.5
>> >> >
>> >> > _______________________________________________
>> >> > meta-arago mailing list
>> >> > meta-arago@arago-project.org
>> >> > http://arago-project.org/cgi-bin/mailman/listinfo/meta-
>arago
>> >> _______________________________________________
>> >> meta-arago mailing list
>> >> meta-arago@arago-project.org
>> >> http://arago-project.org/cgi-bin/mailman/listinfo/meta-arago