From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga14.intel.com ([143.182.124.37]) by linuxtogo.org with esmtp (Exim 4.72) (envelope-from ) id 1Rxgq3-0002B5-8g for openembedded-core@lists.openembedded.org; Wed, 15 Feb 2012 16:32:12 +0100 Received: from azsmga001.ch.intel.com ([10.2.17.19]) by azsmga102.ch.intel.com with ESMTP; 15 Feb 2012 07:23:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.71,315,1320652800"; d="scan'208";a="107068722" Received: from unknown (HELO [10.255.15.43]) ([10.255.15.43]) by azsmga001.ch.intel.com with ESMTP; 15 Feb 2012 07:23:57 -0800 Message-ID: <4F3BCE0C.10400@linux.intel.com> Date: Wed, 15 Feb 2012 07:23:56 -0800 From: Saul Wold User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20120131 Thunderbird/10.0 MIME-Version: 1.0 To: Saul Wold References: <4F21BE12.3000008@linux.intel.com> <4F24F672.5090107@linux.intel.com> <4F38E8CB.6090101@windriver.com> <4F3A9118.8050806@linux.intel.com> In-Reply-To: <4F3A9118.8050806@linux.intel.com> Cc: Patches and discussions about the oe-core layer Subject: Re: [oe] Source Archiver Class X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Patches and discussions about the oe-core layer List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2012 15:32:12 -0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Xiaofeng, I replied last night to your IRC, but was not sure if you saw it, I am adding it to this email. (11:05:36 PM) sgw: xiaofeng, you around? (11:08:22 PM) sgw: zlib is a bad example, since there are no patches! (11:08:38 PM) sgw: Let's use "at" since it is a simple recipe and has patches (11:09:27 PM) sgw: if you start with bitbake -c cleanall at, so we have a fresh start. (11:12:26 PM) sgw: bitbake -c unpack at, will give you the upacked source in at-3.1.12-r7/at-3.1.12/..., this will give you a base to build from (11:15:12 PM) sgw: "bitbake -c patch at" will then patch the above sources and create the patches directory with the series file. So given that info, you be able to add a do_patch[prefunc] to first tar the unpatched directory and then a do_patch[postfunc] to tar the patches directory into the tar file from the first tar. (11:16:32 PM) sgw: Does this make sense? (11:17:49 PM) sgw: Something else we will want to consider is creating one tarball of all the gnu source and patches in one large tarball. On 02/14/2012 08:51 AM, Saul Wold wrote: > On 02/13/2012 02:41 AM, Xiaofeng Yan wrote: >> Hi Saul, >> >> I have some issues when I writing design document. The following >> description is my understanding. I take package "zlib" for example . >>> >>> >>> This is a progression list of what the source archives should include: >>> >>> 1 - Original Upstream Archive & Patches >>> - 2 archives (tarballs) >> $ls zlib-orgsource >> zlib-1.2.5.tar.bz2(after do_fetch) >> zlib-patches.tar.bz2( the patches come from >> meta/recipes-core/zlib/zlib-1.2.5) > > Almost correct, I believe that the work Bruce and Chris have done > recently will allow you to get the patches. > > I am not sure we want to just tar up the > meta/recipes-core/zlib/lib-1.2.5 directory since it might contain other > file or patches that we don't actaully use. We need to get the patches > that are actually applied. > >>> 2 - Original Source code & Patches >>> - could include additional code >> Please give me more detailed information > One tarball with the patches extracted in the patches directory with the > series file (possibly a script to apply the patches). > >>> - post unpack >>> - 1 archive >> $ls zlib-orgsource >> zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this >> package.) >> >> No patch in this directory. > Need to add the patches via patch mechanism to the source tarball, but > without actually applying the patches. > > So, this is just one tarball. > zlib-1.2.5-prepatch.tar.bz2 >>> >>> 3 - Original Source code & Patches & temp (scripts & logs) >>> - Could possibly include the .bb and .inc files >>> - 2 archives (from #2 & temp tarball) >> $ls zlib-orgsource >> zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this >> package.) >> zlib-patches.tar.bz2( the patches come from >> meta/recipes-core/zlib/zlib-1.2.5) > > Again see above, we need to get only the patches that will actually be > applied. > >> zlib-scripts-logs.tar.bz2(include zlib_1.2.5.bb,zlib.inc and tmp/log_*) >> >> So there are 3 tarballs in this directory. > Yes > >>> 4 - Patched source code >>> - original source code could be removed >>> - post do_patch >> $ls zlib-source >> zlib-1.2.5.tar.bz2(after do_patch) > Correct > Maybe better zlib-1.2.5-patched.tar.bz2 > >>> 5 - Patched source code & temp >>> - 2 archives (from #4 & temp tarball) >> $ls zlib-source >> zlib-1.2.5.tar.bz2(after do_patch) >> zlib-logs.tar.bz2(include tmp/log_*) > Correct > Same as #4 change the tarball name > >>> 6 - Configured Source >>> - post do_configure >> $ls zlib-source >> zlib-1.2.5.tar.bz2(after do_configure) > zlib-1.2.5-configured.tar.bz2 > >>> 7 - SRPM format of Original Source & Patches >>> - rpm will apply the patches >>> - internally contains #2 above >> $ls zlib-srpm >> zlib-1.2.5.src.rpm >> >> zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches >> are not included in this package.) and zlib-patches.tar.bz2 >>> 8 - SRPM various of #3 above >> $ls zlib-srpm >> zlib-1.2.5.src.rpm >> >> zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches >> are not included in this package.) ,zlib-patches.tar.bz2 and >> zlib-scripts-logs.tar.bz2 >> >> Do you have any suggestion about the above description? >> > Just the naming and where the patches come from we need to provide the > mechanism (series file) for how to apply the patches also. > > Sau! > >> Thanks >> Yan >> >> >>> ... >>> 100 - Buildable SRPM >>> - can actually build, this is way future! >>> >>> (Patches = patch files & series list) >>> >>> Each of these build on the previous in some way, the key being that we >>> generate a tarball for the source from the existing state of the >>> WORKDIR, a challenge maybe to create the source snapshot after the build >>> has already occurred. >>> >>> The SPDX License info could also be included in any of these >>> >>> After talking with Richard, we think we have a novel approach to make >>> this work. It would entail using the postfuncs feature similar to the >>> way that Shared State does it's work. The existing copyleft_compliance >>> class functionality can be folded into this as a filter. Additional >>> flags could be passed from individual classes to a core set of methods >>> in an archiver class defining the type of data (source, patches, temp, >>> env, recipe info, ...) and format (tar, sprm, ...) >>> >>> I noted that recipe type code might be better suited as generic code, as >>> I believe there are other places that could benefit from that code. >>> >>> The sourcepkg class seems to be a basic archiver and differ that >>> includes the metadata/environment (as dumpdata), this could be replaced >>> by the new approach. While the src_distribute class copies the >>> downloaded archive and then creates a link, into LICENSE directories >>> along with the patches. The src_distribute by default seems to move >>> files and create links (incorrectly btw!). This work can be done by the >>> archiver class. >>> >>> The Nugget: Create a new core "archiver" class that implements a general >>> functions that can archive the original tarball or workdir at various >>> states along task list with additional metadata (recipe info, temp dir, >>> environment). This class would be inherited by a set of classes that use >>> the postfunc (similar to sstate) that setup what level of archive is >>> needed (based on the list above). >>> >>> Thoughts, Comments? >>> >>> Thanks >>> >>> >>> >> >> >