[RFC PULL] Bibliography URL cleanup

[RFC PULL] Bibliography URL cleanup

[Akira Yokosawa]

Hi Paul,

On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
> On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
[snip]
>> So, these bib files are an library collected for nearly three decades!!!
>> They are invaluable as they are, and I'd appreciate your decision to
>> make them public.
> 
> Unfortunately, many of the comments on the early entries reflect my
> relative youth and impetuosity, so unless or until I get time to edit
> the whole mess so as to avoid offending any number of authors (to say
> nothing of their disciples!), I must keep the originals private.

I see. I misunderstood the circumstances. So you made only a part of your
bib files public.

> 
>> There are two issues in urls in the bib files.
>> One is the inconsistency of format discussed here.
>> The other is the dead links. There are quite a few urls that end up in
>> "not found" now. Maintaining urls would require a great deal of work itself...
>>
>> To make the format consistent, a script would work. But before beginning
>> implementation, we need to clarify what the script would do.
>> So I'll make some sample replacement patches to confirm your preference.
> 
> Sounds good, and I look forward to seeing them!

I said to make "some sample replacement patches", but it became quite
intensive changes. So I'm sending them as a pull request. I don't expect
you to actually pull them as it is, but just to pull them on a local
branch and see what they look like.

>							Thanx, Paul
> 

This request consists of 25 patches. Patches 1 and 2 are improvements of
build scripts to make sure that necessary round of pdflatex is run when
only contents of bibliography are modified.

Patches 3 ("bib: Add missing punctuation in 'url' field") to 9 ("bib:
Remove domain part in doi fields") (except for patch 7) are prerequisite
fixes of bib files to be properly parsed with "alphapf" bibliographystyle,
which is a customized version of standard "alpha" style, to be added in
the following patches. The customization is done by "urlbst" tool provided
in TeX Live.

Patch 7 ("Load 'url' package with 'hyphens' option") is not a fix but gives
room for line breaks within urls.

Patches 10 ("Localize alpha.bst") to 13 ("Use 'alphapf' bibliographystyle
instead of 'alpha'") actually replace bibliographystyle.

Patch 14 ("bib/RCU: Shorten author list of 'Appavoo03a'") is obviously
a workaround. The symptom appears only when "inlinelinks" option of
alphapf.bst is enabled. The root cause of the TeX error is not figured
out yet. Once it is fixed this patch can be reverted.

Patch 15 ("alphapf.bst: Enable 'inlinelinks'") does what the title says.

Patches 16--25 do cleanup of bib/realtime.bib. I selected it because
it contains 48 urls which seemed to be a reasonable number for a trial
patch.

Patch 16 ("bib/realtime: Replace 'Available: ... [Viewed ...]' with
'URL: ...'") does what the (not yet implemented) script I mentioned
earlier would do.

Patches 17 ("bib/realtime: Update url of 'BillInmon2007a'") to
22 ("bib/realtime: Update url of 'StephenShankland20Sep2006'") salvage
some of broken urls.

Patch 23 ("bib/realtime: Mark broken urls as such") marks those urls which
could not be salvaged. You may have other opinion of the form of notice
"[broken, ...]" appended to the de-hyperrefed url.

Patch 24 ("bib/realtime: Use alternative url for
'IBMRealTimeJavaTechnology2007a'") replaces a missing url with what seems
to be close to the site originally cited.

Finally, patch 25 ("bib/realtime: Update 'lastchecked' fields") updates
"lastchecked" fields of urls which is reachable.

Let me explain "inlinelinks" option of "alphapf.bst" style (provided by
"urlbst" tool) a little.

When this option is disabled, urls and dois given in corresponding fields
are explicitly printed in Bibliography. In this case, urls are prefixed
by "URL: " by default.  The string is customizable.  But this looks too
verbose for me.

When this option is enabled, they are embedded as hyperlinks of "title"
strings of the entries. This will generate identical output in print as
standard "alpha" style. When both url and doi is provided in an entry, 
doi has a higher priority to be embedded as a hyperlink.
"alphapf.bst" also defines a field named "lastchecked", which is to be
used to indicate when the url is cited.

Regarding these features of alphapf.bst, I'm suggesting the following
entry formats in .bib files.

For "unpublished" entries,

> @unpublished{DavidAWheeler1996
> ,Author="David A. Wheeler"
> ,Title="Ada, C, C++, and Java vs. The Steelman"
> ,year="1996"
> ,note="URL:
> \url{http://www.adahome.com/History/Steelman/steeltab.htm}
> "
> ,lastchecked="November 4, 2016"
> }

The string "URL: " at the beginning of "note" field corresponds to the
default prefix of url printed when "inlinelinks" option is disabled.
You might feel hesitation in directly putting a string which is
customizable elsewhere (in alphapf.bst). It is possible to define a macro
and use it instead in bib entries, but that would cause trouble when you
do the same changes in your private bib library to be used other than
perfbook. So I directly put the string there. If it is all right to use
a macro, please let me know. I'll do a respin or add a patch just for
the replacement.

For other types of entries such as "conference",

> @conference{PeterOkech2009InherentRandomness
> ,Author="Nicholas {Mc Guire} and Peter Odhiambo Okech and Qingguo Zhou"
> ,Title="Analysis of inherent randomness of the Linux kernel"
> ,Booktitle="Eleventh Real Time Linux Workshop"
> ,month="September"
> ,year="2009"
> ,address="Dresden, Germany"
> ,url={https://www.osadl.org/?id=684}
> ,lastchecked="November 4, 2016"
> }

if you don't want the url to be printed in Bibliography.

Or,

> @conference{JoshTriplett2009PainlessKernel
> ,Author="Josh Triplett"
> ,Title="Painless kernel - removing the {HZ}"
> ,Booktitle="Linux Plumbers Conference"
> ,month="September"
> ,year="2009"
> ,address="Portland, OR, USA"
> ,note="URL:
> \url{http://linuxplumbersconf.org/2009/slides/Josh-Triplett-painless-kernel.pdf}"
> ,lastchecked="November 4, 2016"
> }

if you want the url to be printed.

Dates given in "lastchecked" fields are printed in the form of [cited ...]
when "inlinelinks" option is disabled and both "url" and "lastchecked" fields
exist in an entry. The string "cited " is customizable.

Also, if doi is available, it is expected to be stabler and more preferable than
a raw url.  This type of change is done in patch 21 ("bib/realtime: Replace url
with doi for 'RobertBerry2008IBMSysJ'"). The result is as follows:

> @article{RobertBerry2008IBMSysJ
> ,author="R. F. Berry and P. E. McKenney and F. N. Parr"
> ,title="Responsive systems: An introduction"
> ,Year="2008"
> ,Month="April"
> ,journal="IBM Systems Journal"
> ,volume="47"
> ,number="2"
> ,pages="197-206"
> ,doi="10.1147/sj.472.0197"
> }

Both "doi" and "url" fields can be given in an entry. 

As for broken links, I'm suggesting the following format:

> @unpublished{KristofferBohmann2001a
> ,Author="Kristoffer Bohmann"
> ,Title="Response Time Still Matters"
> ,month="July"
> ,year="2001"
> ,day="12"
> ,note="URL:
> \nolinkurl{http://www.bohmann.dk/articles/response_time_still_matters.html}
> [broken, November 2016]"
> ,lastchecked="July 23, 2007"
> }

This keeps the original "Viewed" date in "lastchecked" field.
Url is de-hyperrefed within \nolinkurl{} command.
If it becomes clear the content is not recoverable, you might want to remove
or modify text where it is cited.

The bad news for the cleanup is that there are a variety of format of "note"
fields found in other .bib files, and it seems not easy to implement a script
to do changes as patch 16 which covers all the cases. It might be easier to
manually edit by using keyboard macro of emacs...

Anyway, following is the pull request of the changes. Please take your time
to see and let me know what you think.

FYI, you might want to pull up to patch 9 ("bib: Remove domain part in doi
fields"). They are improvements and (potential) bug fixes.

                                            Thanks, Akira
----
The following changes since commit bebc538fe4ee24603936e31c981e5342f85b88e5:

  Fix several typos (2016-10-26 16:15:36 -0700)

are available in the git repository at:

  https://github.com/akiyks/perfbook.git bib-url-cleanup-v1

for you to fetch changes up to 1b30f5f91a9bdd133c85d59b41201881b49b8872:

  bib/realtime: Update 'lastchecked' fields (2016-11-05 09:23:25 +0900)

----------------------------------------------------------------
Akira Yokosawa (25):
      runlatex.sh: Add a round for possible bib update
      Makefile: Move $(BIBSOURCES) to dependency of .aux target
      bib: Add missing punctuation in 'url' field
      bib: Fix errors around \url{} command
      bib: Remove nested \url{} in 'url' field
      bib: Add missing \url{} command
      Load 'url' package with 'hyphens' option
      bib/os: Enclose url of 'BenjaminGamsa95a' in \url{} command
      bib: Remove domain part in doi fields
      Localize alpha.bst
      Costomize alpha.bst by 'urlbst' and rename as alphapf.bst
      alphapf.bst: Reorder 'note' field of 'unpublished' entry
      Use 'alphapf' bibliographystyle instead of 'alpha'
      bib/RCU: Shorten author list of 'Appavoo03a'
      alphapf.bst: Enable 'inlinelinks'
      bib/realtime: Replace 'Available: ... [Viewed ...]' with 'URL: ...'
      bib/realtime: Update url of 'BillInmon2007a'
      bib/realtime: Update url of 'KelvinNilsen2007'
      bib/realtime: Replace url of 'PaulEMcKenney2008OLS'
      bib/realtime: Update url of 'SunMicrosystems2008RTSJavaGC'
      bib/realtime: Replace url with doi for 'RobertBerry2008IBMSysJ'
      bib/realtime: Update url of 'StephenShankland20Sep2006'
      bib/realtime: Mark broken urls as such
      bib/realtime: Use alternative url for 'IBMRealTimeJavaTechnology2007a'
      bib/realtime: Update 'lastchecked' fields

 Makefile              |    6 +-
 alphapf.bst           | 1613 +++++++++++++++++++++++++++++++++++++++++++++++++
 appendix/appendix.tex |    2 +-
 bib/RCU.bib           |   27 +-
 bib/RCUuses.bib       |    2 +-
 bib/TM.bib            |   39 +-
 bib/WFS.bib           |   19 +-
 bib/energy.bib        |    4 +-
 bib/hw.bib            |    2 +-
 bib/os.bib            |    6 +-
 bib/parallelsys.bib   |    8 +-
 bib/realtime.bib      |  264 ++++----
 bib/swtools.bib       |   12 +-
 bib/syncrefs.bib      |   10 +-
 perfbook.tex          |    1 +
 utilities/runlatex.sh |   10 +-
 16 files changed, 1851 insertions(+), 174 deletions(-)
 create mode 100644 alphapf.bst

Re: [RFC PULL] Bibliography URL cleanup

[Paul E. McKenney]

On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
> Hi Paul,
> 
> On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
> > On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
> [snip]
> >> So, these bib files are an library collected for nearly three decades!!!
> >> They are invaluable as they are, and I'd appreciate your decision to
> >> make them public.
> > 
> > Unfortunately, many of the comments on the early entries reflect my
> > relative youth and impetuosity, so unless or until I get time to edit
> > the whole mess so as to avoid offending any number of authors (to say
> > nothing of their disciples!), I must keep the originals private.
> 
> I see. I misunderstood the circumstances. So you made only a part of your
> bib files public.
> 
> > 
> >> There are two issues in urls in the bib files.
> >> One is the inconsistency of format discussed here.
> >> The other is the dead links. There are quite a few urls that end up in
> >> "not found" now. Maintaining urls would require a great deal of work itself...
> >>
> >> To make the format consistent, a script would work. But before beginning
> >> implementation, we need to clarify what the script would do.
> >> So I'll make some sample replacement patches to confirm your preference.
> > 
> > Sounds good, and I look forward to seeing them!
> 
> I said to make "some sample replacement patches", but it became quite
> intensive changes. So I'm sending them as a pull request. I don't expect
> you to actually pull them as it is, but just to pull them on a local
> branch and see what they look like.

I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
patches.  I am reflecting those changes in my bib sources as well.
Looked sane at first glance, but yes, I need to work out how to handle
the later ones with other documents...

							Thanx, Paul

> This request consists of 25 patches. Patches 1 and 2 are improvements of
> build scripts to make sure that necessary round of pdflatex is run when
> only contents of bibliography are modified.
> 
> Patches 3 ("bib: Add missing punctuation in 'url' field") to 9 ("bib:
> Remove domain part in doi fields") (except for patch 7) are prerequisite
> fixes of bib files to be properly parsed with "alphapf" bibliographystyle,
> which is a customized version of standard "alpha" style, to be added in
> the following patches. The customization is done by "urlbst" tool provided
> in TeX Live.
> 
> Patch 7 ("Load 'url' package with 'hyphens' option") is not a fix but gives
> room for line breaks within urls.
> 
> Patches 10 ("Localize alpha.bst") to 13 ("Use 'alphapf' bibliographystyle
> instead of 'alpha'") actually replace bibliographystyle.
> 
> Patch 14 ("bib/RCU: Shorten author list of 'Appavoo03a'") is obviously
> a workaround. The symptom appears only when "inlinelinks" option of
> alphapf.bst is enabled. The root cause of the TeX error is not figured
> out yet. Once it is fixed this patch can be reverted.
> 
> Patch 15 ("alphapf.bst: Enable 'inlinelinks'") does what the title says.
> 
> Patches 16--25 do cleanup of bib/realtime.bib. I selected it because
> it contains 48 urls which seemed to be a reasonable number for a trial
> patch.
> 
> Patch 16 ("bib/realtime: Replace 'Available: ... [Viewed ...]' with
> 'URL: ...'") does what the (not yet implemented) script I mentioned
> earlier would do.
> 
> Patches 17 ("bib/realtime: Update url of 'BillInmon2007a'") to
> 22 ("bib/realtime: Update url of 'StephenShankland20Sep2006'") salvage
> some of broken urls.
> 
> Patch 23 ("bib/realtime: Mark broken urls as such") marks those urls which
> could not be salvaged. You may have other opinion of the form of notice
> "[broken, ...]" appended to the de-hyperrefed url.
> 
> Patch 24 ("bib/realtime: Use alternative url for
> 'IBMRealTimeJavaTechnology2007a'") replaces a missing url with what seems
> to be close to the site originally cited.
> 
> Finally, patch 25 ("bib/realtime: Update 'lastchecked' fields") updates
> "lastchecked" fields of urls which is reachable.
> 
> Let me explain "inlinelinks" option of "alphapf.bst" style (provided by
> "urlbst" tool) a little.
> 
> When this option is disabled, urls and dois given in corresponding fields
> are explicitly printed in Bibliography. In this case, urls are prefixed
> by "URL: " by default.  The string is customizable.  But this looks too
> verbose for me.
> 
> When this option is enabled, they are embedded as hyperlinks of "title"
> strings of the entries. This will generate identical output in print as
> standard "alpha" style. When both url and doi is provided in an entry, 
> doi has a higher priority to be embedded as a hyperlink.
> "alphapf.bst" also defines a field named "lastchecked", which is to be
> used to indicate when the url is cited.
> 
> Regarding these features of alphapf.bst, I'm suggesting the following
> entry formats in .bib files.
> 
> For "unpublished" entries,
> 
> > @unpublished{DavidAWheeler1996
> > ,Author="David A. Wheeler"
> > ,Title="Ada, C, C++, and Java vs. The Steelman"
> > ,year="1996"
> > ,note="URL:
> > \url{http://www.adahome.com/History/Steelman/steeltab.htm}
> > "
> > ,lastchecked="November 4, 2016"
> > }
> 
> The string "URL: " at the beginning of "note" field corresponds to the
> default prefix of url printed when "inlinelinks" option is disabled.
> You might feel hesitation in directly putting a string which is
> customizable elsewhere (in alphapf.bst). It is possible to define a macro
> and use it instead in bib entries, but that would cause trouble when you
> do the same changes in your private bib library to be used other than
> perfbook. So I directly put the string there. If it is all right to use
> a macro, please let me know. I'll do a respin or add a patch just for
> the replacement.
> 
> For other types of entries such as "conference",
> 
> > @conference{PeterOkech2009InherentRandomness
> > ,Author="Nicholas {Mc Guire} and Peter Odhiambo Okech and Qingguo Zhou"
> > ,Title="Analysis of inherent randomness of the Linux kernel"
> > ,Booktitle="Eleventh Real Time Linux Workshop"
> > ,month="September"
> > ,year="2009"
> > ,address="Dresden, Germany"
> > ,url={https://www.osadl.org/?id=684}
> > ,lastchecked="November 4, 2016"
> > }
> 
> if you don't want the url to be printed in Bibliography.
> 
> Or,
> 
> > @conference{JoshTriplett2009PainlessKernel
> > ,Author="Josh Triplett"
> > ,Title="Painless kernel - removing the {HZ}"
> > ,Booktitle="Linux Plumbers Conference"
> > ,month="September"
> > ,year="2009"
> > ,address="Portland, OR, USA"
> > ,note="URL:
> > \url{http://linuxplumbersconf.org/2009/slides/Josh-Triplett-painless-kernel.pdf}"
> > ,lastchecked="November 4, 2016"
> > }
> 
> if you want the url to be printed.
> 
> Dates given in "lastchecked" fields are printed in the form of [cited ...]
> when "inlinelinks" option is disabled and both "url" and "lastchecked" fields
> exist in an entry. The string "cited " is customizable.
> 
> Also, if doi is available, it is expected to be stabler and more preferable than
> a raw url.  This type of change is done in patch 21 ("bib/realtime: Replace url
> with doi for 'RobertBerry2008IBMSysJ'"). The result is as follows:
> 
> > @article{RobertBerry2008IBMSysJ
> > ,author="R. F. Berry and P. E. McKenney and F. N. Parr"
> > ,title="Responsive systems: An introduction"
> > ,Year="2008"
> > ,Month="April"
> > ,journal="IBM Systems Journal"
> > ,volume="47"
> > ,number="2"
> > ,pages="197-206"
> > ,doi="10.1147/sj.472.0197"
> > }
> 
> Both "doi" and "url" fields can be given in an entry. 
> 
> As for broken links, I'm suggesting the following format:
> 
> > @unpublished{KristofferBohmann2001a
> > ,Author="Kristoffer Bohmann"
> > ,Title="Response Time Still Matters"
> > ,month="July"
> > ,year="2001"
> > ,day="12"
> > ,note="URL:
> > \nolinkurl{http://www.bohmann.dk/articles/response_time_still_matters.html}
> > [broken, November 2016]"
> > ,lastchecked="July 23, 2007"
> > }
> 
> This keeps the original "Viewed" date in "lastchecked" field.
> Url is de-hyperrefed within \nolinkurl{} command.
> If it becomes clear the content is not recoverable, you might want to remove
> or modify text where it is cited.
> 
> The bad news for the cleanup is that there are a variety of format of "note"
> fields found in other .bib files, and it seems not easy to implement a script
> to do changes as patch 16 which covers all the cases. It might be easier to
> manually edit by using keyboard macro of emacs...
> 
> Anyway, following is the pull request of the changes. Please take your time
> to see and let me know what you think.
> 
> FYI, you might want to pull up to patch 9 ("bib: Remove domain part in doi
> fields"). They are improvements and (potential) bug fixes.
> 
>                                             Thanks, Akira
> ----
> The following changes since commit bebc538fe4ee24603936e31c981e5342f85b88e5:
> 
>   Fix several typos (2016-10-26 16:15:36 -0700)
> 
> are available in the git repository at:
> 
>   https://github.com/akiyks/perfbook.git bib-url-cleanup-v1
> 
> for you to fetch changes up to 1b30f5f91a9bdd133c85d59b41201881b49b8872:
> 
>   bib/realtime: Update 'lastchecked' fields (2016-11-05 09:23:25 +0900)
> 
> ----------------------------------------------------------------
> Akira Yokosawa (25):
>       runlatex.sh: Add a round for possible bib update
>       Makefile: Move $(BIBSOURCES) to dependency of .aux target
>       bib: Add missing punctuation in 'url' field
>       bib: Fix errors around \url{} command
>       bib: Remove nested \url{} in 'url' field
>       bib: Add missing \url{} command
>       Load 'url' package with 'hyphens' option
>       bib/os: Enclose url of 'BenjaminGamsa95a' in \url{} command
>       bib: Remove domain part in doi fields
>       Localize alpha.bst
>       Costomize alpha.bst by 'urlbst' and rename as alphapf.bst
>       alphapf.bst: Reorder 'note' field of 'unpublished' entry
>       Use 'alphapf' bibliographystyle instead of 'alpha'
>       bib/RCU: Shorten author list of 'Appavoo03a'
>       alphapf.bst: Enable 'inlinelinks'
>       bib/realtime: Replace 'Available: ... [Viewed ...]' with 'URL: ...'
>       bib/realtime: Update url of 'BillInmon2007a'
>       bib/realtime: Update url of 'KelvinNilsen2007'
>       bib/realtime: Replace url of 'PaulEMcKenney2008OLS'
>       bib/realtime: Update url of 'SunMicrosystems2008RTSJavaGC'
>       bib/realtime: Replace url with doi for 'RobertBerry2008IBMSysJ'
>       bib/realtime: Update url of 'StephenShankland20Sep2006'
>       bib/realtime: Mark broken urls as such
>       bib/realtime: Use alternative url for 'IBMRealTimeJavaTechnology2007a'
>       bib/realtime: Update 'lastchecked' fields
> 
>  Makefile              |    6 +-
>  alphapf.bst           | 1613 +++++++++++++++++++++++++++++++++++++++++++++++++
>  appendix/appendix.tex |    2 +-
>  bib/RCU.bib           |   27 +-
>  bib/RCUuses.bib       |    2 +-
>  bib/TM.bib            |   39 +-
>  bib/WFS.bib           |   19 +-
>  bib/energy.bib        |    4 +-
>  bib/hw.bib            |    2 +-
>  bib/os.bib            |    6 +-
>  bib/parallelsys.bib   |    8 +-
>  bib/realtime.bib      |  264 ++++----
>  bib/swtools.bib       |   12 +-
>  bib/syncrefs.bib      |   10 +-
>  perfbook.tex          |    1 +
>  utilities/runlatex.sh |   10 +-
>  16 files changed, 1851 insertions(+), 174 deletions(-)
>  create mode 100644 alphapf.bst
> 

Re: [RFC PULL] Bibliography URL cleanup

[Paul E. McKenney]

On Sun, Nov 06, 2016 at 03:47:19PM -0800, Paul E. McKenney wrote:
> On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
> > Hi Paul,
> > 
> > On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
> > > On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
> > [snip]
> > >> So, these bib files are an library collected for nearly three decades!!!
> > >> They are invaluable as they are, and I'd appreciate your decision to
> > >> make them public.
> > > 
> > > Unfortunately, many of the comments on the early entries reflect my
> > > relative youth and impetuosity, so unless or until I get time to edit
> > > the whole mess so as to avoid offending any number of authors (to say
> > > nothing of their disciples!), I must keep the originals private.
> > 
> > I see. I misunderstood the circumstances. So you made only a part of your
> > bib files public.
> > 
> > > 
> > >> There are two issues in urls in the bib files.
> > >> One is the inconsistency of format discussed here.
> > >> The other is the dead links. There are quite a few urls that end up in
> > >> "not found" now. Maintaining urls would require a great deal of work itself...
> > >>
> > >> To make the format consistent, a script would work. But before beginning
> > >> implementation, we need to clarify what the script would do.
> > >> So I'll make some sample replacement patches to confirm your preference.
> > > 
> > > Sounds good, and I look forward to seeing them!
> > 
> > I said to make "some sample replacement patches", but it became quite
> > intensive changes. So I'm sending them as a pull request. I don't expect
> > you to actually pull them as it is, but just to pull them on a local
> > branch and see what they look like.
> 
> I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
> patches.  I am reflecting those changes in my bib sources as well.
> Looked sane at first glance, but yes, I need to work out how to handle
> the later ones with other documents...

And the later .bib edits seem compatible with my current tools, even
without the script and .bst changes.  So I have applied them in tandem
to my .bib source and to the perfbook bibliography.  I was able to
find valid URLs for a few of the entries, so used them instead of
\nolinkurl{}, but several do appear to be quite dead.

I also applied the alphapf.bst changes, but left inlinelinks disabled
for the time being.  (I am concerned about leaving authors off.)

Thank you very much for your work on this!!!

							Thanx, Paul

> > This request consists of 25 patches. Patches 1 and 2 are improvements of
> > build scripts to make sure that necessary round of pdflatex is run when
> > only contents of bibliography are modified.
> > 
> > Patches 3 ("bib: Add missing punctuation in 'url' field") to 9 ("bib:
> > Remove domain part in doi fields") (except for patch 7) are prerequisite
> > fixes of bib files to be properly parsed with "alphapf" bibliographystyle,
> > which is a customized version of standard "alpha" style, to be added in
> > the following patches. The customization is done by "urlbst" tool provided
> > in TeX Live.
> > 
> > Patch 7 ("Load 'url' package with 'hyphens' option") is not a fix but gives
> > room for line breaks within urls.
> > 
> > Patches 10 ("Localize alpha.bst") to 13 ("Use 'alphapf' bibliographystyle
> > instead of 'alpha'") actually replace bibliographystyle.
> > 
> > Patch 14 ("bib/RCU: Shorten author list of 'Appavoo03a'") is obviously
> > a workaround. The symptom appears only when "inlinelinks" option of
> > alphapf.bst is enabled. The root cause of the TeX error is not figured
> > out yet. Once it is fixed this patch can be reverted.
> > 
> > Patch 15 ("alphapf.bst: Enable 'inlinelinks'") does what the title says.
> > 
> > Patches 16--25 do cleanup of bib/realtime.bib. I selected it because
> > it contains 48 urls which seemed to be a reasonable number for a trial
> > patch.
> > 
> > Patch 16 ("bib/realtime: Replace 'Available: ... [Viewed ...]' with
> > 'URL: ...'") does what the (not yet implemented) script I mentioned
> > earlier would do.
> > 
> > Patches 17 ("bib/realtime: Update url of 'BillInmon2007a'") to
> > 22 ("bib/realtime: Update url of 'StephenShankland20Sep2006'") salvage
> > some of broken urls.
> > 
> > Patch 23 ("bib/realtime: Mark broken urls as such") marks those urls which
> > could not be salvaged. You may have other opinion of the form of notice
> > "[broken, ...]" appended to the de-hyperrefed url.
> > 
> > Patch 24 ("bib/realtime: Use alternative url for
> > 'IBMRealTimeJavaTechnology2007a'") replaces a missing url with what seems
> > to be close to the site originally cited.
> > 
> > Finally, patch 25 ("bib/realtime: Update 'lastchecked' fields") updates
> > "lastchecked" fields of urls which is reachable.
> > 
> > Let me explain "inlinelinks" option of "alphapf.bst" style (provided by
> > "urlbst" tool) a little.
> > 
> > When this option is disabled, urls and dois given in corresponding fields
> > are explicitly printed in Bibliography. In this case, urls are prefixed
> > by "URL: " by default.  The string is customizable.  But this looks too
> > verbose for me.
> > 
> > When this option is enabled, they are embedded as hyperlinks of "title"
> > strings of the entries. This will generate identical output in print as
> > standard "alpha" style. When both url and doi is provided in an entry, 
> > doi has a higher priority to be embedded as a hyperlink.
> > "alphapf.bst" also defines a field named "lastchecked", which is to be
> > used to indicate when the url is cited.
> > 
> > Regarding these features of alphapf.bst, I'm suggesting the following
> > entry formats in .bib files.
> > 
> > For "unpublished" entries,
> > 
> > > @unpublished{DavidAWheeler1996
> > > ,Author="David A. Wheeler"
> > > ,Title="Ada, C, C++, and Java vs. The Steelman"
> > > ,year="1996"
> > > ,note="URL:
> > > \url{http://www.adahome.com/History/Steelman/steeltab.htm}
> > > "
> > > ,lastchecked="November 4, 2016"
> > > }
> > 
> > The string "URL: " at the beginning of "note" field corresponds to the
> > default prefix of url printed when "inlinelinks" option is disabled.
> > You might feel hesitation in directly putting a string which is
> > customizable elsewhere (in alphapf.bst). It is possible to define a macro
> > and use it instead in bib entries, but that would cause trouble when you
> > do the same changes in your private bib library to be used other than
> > perfbook. So I directly put the string there. If it is all right to use
> > a macro, please let me know. I'll do a respin or add a patch just for
> > the replacement.
> > 
> > For other types of entries such as "conference",
> > 
> > > @conference{PeterOkech2009InherentRandomness
> > > ,Author="Nicholas {Mc Guire} and Peter Odhiambo Okech and Qingguo Zhou"
> > > ,Title="Analysis of inherent randomness of the Linux kernel"
> > > ,Booktitle="Eleventh Real Time Linux Workshop"
> > > ,month="September"
> > > ,year="2009"
> > > ,address="Dresden, Germany"
> > > ,url={https://www.osadl.org/?id=684}
> > > ,lastchecked="November 4, 2016"
> > > }
> > 
> > if you don't want the url to be printed in Bibliography.
> > 
> > Or,
> > 
> > > @conference{JoshTriplett2009PainlessKernel
> > > ,Author="Josh Triplett"
> > > ,Title="Painless kernel - removing the {HZ}"
> > > ,Booktitle="Linux Plumbers Conference"
> > > ,month="September"
> > > ,year="2009"
> > > ,address="Portland, OR, USA"
> > > ,note="URL:
> > > \url{http://linuxplumbersconf.org/2009/slides/Josh-Triplett-painless-kernel.pdf}"
> > > ,lastchecked="November 4, 2016"
> > > }
> > 
> > if you want the url to be printed.
> > 
> > Dates given in "lastchecked" fields are printed in the form of [cited ...]
> > when "inlinelinks" option is disabled and both "url" and "lastchecked" fields
> > exist in an entry. The string "cited " is customizable.
> > 
> > Also, if doi is available, it is expected to be stabler and more preferable than
> > a raw url.  This type of change is done in patch 21 ("bib/realtime: Replace url
> > with doi for 'RobertBerry2008IBMSysJ'"). The result is as follows:
> > 
> > > @article{RobertBerry2008IBMSysJ
> > > ,author="R. F. Berry and P. E. McKenney and F. N. Parr"
> > > ,title="Responsive systems: An introduction"
> > > ,Year="2008"
> > > ,Month="April"
> > > ,journal="IBM Systems Journal"
> > > ,volume="47"
> > > ,number="2"
> > > ,pages="197-206"
> > > ,doi="10.1147/sj.472.0197"
> > > }
> > 
> > Both "doi" and "url" fields can be given in an entry. 
> > 
> > As for broken links, I'm suggesting the following format:
> > 
> > > @unpublished{KristofferBohmann2001a
> > > ,Author="Kristoffer Bohmann"
> > > ,Title="Response Time Still Matters"
> > > ,month="July"
> > > ,year="2001"
> > > ,day="12"
> > > ,note="URL:
> > > \nolinkurl{http://www.bohmann.dk/articles/response_time_still_matters.html}
> > > [broken, November 2016]"
> > > ,lastchecked="July 23, 2007"
> > > }
> > 
> > This keeps the original "Viewed" date in "lastchecked" field.
> > Url is de-hyperrefed within \nolinkurl{} command.
> > If it becomes clear the content is not recoverable, you might want to remove
> > or modify text where it is cited.
> > 
> > The bad news for the cleanup is that there are a variety of format of "note"
> > fields found in other .bib files, and it seems not easy to implement a script
> > to do changes as patch 16 which covers all the cases. It might be easier to
> > manually edit by using keyboard macro of emacs...
> > 
> > Anyway, following is the pull request of the changes. Please take your time
> > to see and let me know what you think.
> > 
> > FYI, you might want to pull up to patch 9 ("bib: Remove domain part in doi
> > fields"). They are improvements and (potential) bug fixes.
> > 
> >                                             Thanks, Akira
> > ----
> > The following changes since commit bebc538fe4ee24603936e31c981e5342f85b88e5:
> > 
> >   Fix several typos (2016-10-26 16:15:36 -0700)
> > 
> > are available in the git repository at:
> > 
> >   https://github.com/akiyks/perfbook.git bib-url-cleanup-v1
> > 
> > for you to fetch changes up to 1b30f5f91a9bdd133c85d59b41201881b49b8872:
> > 
> >   bib/realtime: Update 'lastchecked' fields (2016-11-05 09:23:25 +0900)
> > 
> > ----------------------------------------------------------------
> > Akira Yokosawa (25):
> >       runlatex.sh: Add a round for possible bib update
> >       Makefile: Move $(BIBSOURCES) to dependency of .aux target
> >       bib: Add missing punctuation in 'url' field
> >       bib: Fix errors around \url{} command
> >       bib: Remove nested \url{} in 'url' field
> >       bib: Add missing \url{} command
> >       Load 'url' package with 'hyphens' option
> >       bib/os: Enclose url of 'BenjaminGamsa95a' in \url{} command
> >       bib: Remove domain part in doi fields
> >       Localize alpha.bst
> >       Costomize alpha.bst by 'urlbst' and rename as alphapf.bst
> >       alphapf.bst: Reorder 'note' field of 'unpublished' entry
> >       Use 'alphapf' bibliographystyle instead of 'alpha'
> >       bib/RCU: Shorten author list of 'Appavoo03a'
> >       alphapf.bst: Enable 'inlinelinks'
> >       bib/realtime: Replace 'Available: ... [Viewed ...]' with 'URL: ...'
> >       bib/realtime: Update url of 'BillInmon2007a'
> >       bib/realtime: Update url of 'KelvinNilsen2007'
> >       bib/realtime: Replace url of 'PaulEMcKenney2008OLS'
> >       bib/realtime: Update url of 'SunMicrosystems2008RTSJavaGC'
> >       bib/realtime: Replace url with doi for 'RobertBerry2008IBMSysJ'
> >       bib/realtime: Update url of 'StephenShankland20Sep2006'
> >       bib/realtime: Mark broken urls as such
> >       bib/realtime: Use alternative url for 'IBMRealTimeJavaTechnology2007a'
> >       bib/realtime: Update 'lastchecked' fields
> > 
> >  Makefile              |    6 +-
> >  alphapf.bst           | 1613 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  appendix/appendix.tex |    2 +-
> >  bib/RCU.bib           |   27 +-
> >  bib/RCUuses.bib       |    2 +-
> >  bib/TM.bib            |   39 +-
> >  bib/WFS.bib           |   19 +-
> >  bib/energy.bib        |    4 +-
> >  bib/hw.bib            |    2 +-
> >  bib/os.bib            |    6 +-
> >  bib/parallelsys.bib   |    8 +-
> >  bib/realtime.bib      |  264 ++++----
> >  bib/swtools.bib       |   12 +-
> >  bib/syncrefs.bib      |   10 +-
> >  perfbook.tex          |    1 +
> >  utilities/runlatex.sh |   10 +-
> >  16 files changed, 1851 insertions(+), 174 deletions(-)
> >  create mode 100644 alphapf.bst
> > 

Re: [RFC PULL] Bibliography URL cleanup

[Akira Yokosawa]

On 2016/12/24 23:05:05 -0800, Paul E. McKenney wrote:
> On Sun, Nov 06, 2016 at 03:47:19PM -0800, Paul E. McKenney wrote:
>> On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
>>> Hi Paul,
>>>
>>> On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
>>>> On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
>>> [snip]
>>>>> So, these bib files are an library collected for nearly three decades!!!
>>>>> They are invaluable as they are, and I'd appreciate your decision to
>>>>> make them public.
>>>>
>>>> Unfortunately, many of the comments on the early entries reflect my
>>>> relative youth and impetuosity, so unless or until I get time to edit
>>>> the whole mess so as to avoid offending any number of authors (to say
>>>> nothing of their disciples!), I must keep the originals private.
>>>
>>> I see. I misunderstood the circumstances. So you made only a part of your
>>> bib files public.
>>>
>>>>
>>>>> There are two issues in urls in the bib files.
>>>>> One is the inconsistency of format discussed here.
>>>>> The other is the dead links. There are quite a few urls that end up in
>>>>> "not found" now. Maintaining urls would require a great deal of work itself...
>>>>>
>>>>> To make the format consistent, a script would work. But before beginning
>>>>> implementation, we need to clarify what the script would do.
>>>>> So I'll make some sample replacement patches to confirm your preference.
>>>>
>>>> Sounds good, and I look forward to seeing them!
>>>
>>> I said to make "some sample replacement patches", but it became quite
>>> intensive changes. So I'm sending them as a pull request. I don't expect
>>> you to actually pull them as it is, but just to pull them on a local
>>> branch and see what they look like.
>>
>> I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
>> patches.  I am reflecting those changes in my bib sources as well.
>> Looked sane at first glance, but yes, I need to work out how to handle
>> the later ones with other documents...
> 
> And the later .bib edits seem compatible with my current tools, even
> without the script and .bst changes.  So I have applied them in tandem
> to my .bib source and to the perfbook bibliography.  I was able to
> find valid URLs for a few of the entries, so used them instead of
> \nolinkurl{}, but several do appear to be quite dead.
> 
> I also applied the alphapf.bst changes, but left inlinelinks disabled
> for the time being.  (I am concerned about leaving authors off.)
> 
> Thank you very much for your work on this!!!

You are welcome!

Now, the branch "bib-append-doi" might have some conflicts.
I'll rebase it and submit a v2 of the PULL request.

inlinelinks seems to have upper limits of around 13 authors.
I've not found any workaround of the issue...
Can't you live with a short author field supplemented by a note of
full authors list?

                                                   Thanks, Akira

> 
> 							Thanx, Paul
> 
[snip]

Re: [RFC PULL] Bibliography URL cleanup

[Akira Yokosawa]

On 2016/12/24 16:36:32 +0900, Akira Yokosawa wrote:
> On 2016/12/24 23:05:05 -0800, Paul E. McKenney wrote:
>> On Sun, Nov 06, 2016 at 03:47:19PM -0800, Paul E. McKenney wrote:
>>> On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
>>>> Hi Paul,
>>>>
>>>> On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
>>>>> On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
>>>> [snip]
>>>>>> So, these bib files are an library collected for nearly three decades!!!
>>>>>> They are invaluable as they are, and I'd appreciate your decision to
>>>>>> make them public.
>>>>>
>>>>> Unfortunately, many of the comments on the early entries reflect my
>>>>> relative youth and impetuosity, so unless or until I get time to edit
>>>>> the whole mess so as to avoid offending any number of authors (to say
>>>>> nothing of their disciples!), I must keep the originals private.
>>>>
>>>> I see. I misunderstood the circumstances. So you made only a part of your
>>>> bib files public.
>>>>
>>>>>
>>>>>> There are two issues in urls in the bib files.
>>>>>> One is the inconsistency of format discussed here.
>>>>>> The other is the dead links. There are quite a few urls that end up in
>>>>>> "not found" now. Maintaining urls would require a great deal of work itself...
>>>>>>
>>>>>> To make the format consistent, a script would work. But before beginning
>>>>>> implementation, we need to clarify what the script would do.
>>>>>> So I'll make some sample replacement patches to confirm your preference.
>>>>>
>>>>> Sounds good, and I look forward to seeing them!
>>>>
>>>> I said to make "some sample replacement patches", but it became quite
>>>> intensive changes. So I'm sending them as a pull request. I don't expect
>>>> you to actually pull them as it is, but just to pull them on a local
>>>> branch and see what they look like.
>>>
>>> I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
>>> patches.  I am reflecting those changes in my bib sources as well.
>>> Looked sane at first glance, but yes, I need to work out how to handle
>>> the later ones with other documents...
>>
>> And the later .bib edits seem compatible with my current tools, even
>> without the script and .bst changes.  So I have applied them in tandem
>> to my .bib source and to the perfbook bibliography.  I was able to
>> find valid URLs for a few of the entries, so used them instead of
>> \nolinkurl{}, but several do appear to be quite dead.
>>
>> I also applied the alphapf.bst changes, but left inlinelinks disabled
>> for the time being.  (I am concerned about leaving authors off.)
>>
>> Thank you very much for your work on this!!!
> 
> You are welcome!
> 
> Now, the branch "bib-append-doi" might have some conflicts.
> I'll rebase it and submit a v2 of the PULL request.

It turns out there is no conflict. So no v2 of the PULL request.

                                                   Thanks, Akira
                                          
> 
> inlinelinks seems to have upper limits of around 13 authors.
> I've not found any workaround of the issue...
> Can't you live with a short author field supplemented by a note of
> full authors list?
> 
>                                                    Thanks, Akira
> 
>>
>> 							Thanx, Paul
>>
> [snip]
> 

[PATCH 0/2] Add script to convert BiBTeX output (was Re: [RFC PULL] Bibliography URL cleanup)

[Akira Yokosawa]

On 2016/12/24 16:36:32 +0900, Akira Yokosawa wrote:
> On 2016/12/24 23:05:05 -0800, Paul E. McKenney wrote:
>> On Sun, Nov 06, 2016 at 03:47:19PM -0800, Paul E. McKenney wrote:
>>> On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
>>>> Hi Paul,
>>>>
>>>> On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
>>>>> On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
>>>> [snip]
>>>>>> So, these bib files are an library collected for nearly three decades!!!
>>>>>> They are invaluable as they are, and I'd appreciate your decision to
>>>>>> make them public.
>>>>>
>>>>> Unfortunately, many of the comments on the early entries reflect my
>>>>> relative youth and impetuosity, so unless or until I get time to edit
>>>>> the whole mess so as to avoid offending any number of authors (to say
>>>>> nothing of their disciples!), I must keep the originals private.
>>>>
>>>> I see. I misunderstood the circumstances. So you made only a part of your
>>>> bib files public.
>>>>
>>>>>
>>>>>> There are two issues in urls in the bib files.
>>>>>> One is the inconsistency of format discussed here.
>>>>>> The other is the dead links. There are quite a few urls that end up in
>>>>>> "not found" now. Maintaining urls would require a great deal of work itself...
>>>>>>
>>>>>> To make the format consistent, a script would work. But before beginning
>>>>>> implementation, we need to clarify what the script would do.
>>>>>> So I'll make some sample replacement patches to confirm your preference.
>>>>>
>>>>> Sounds good, and I look forward to seeing them!
>>>>
>>>> I said to make "some sample replacement patches", but it became quite
>>>> intensive changes. So I'm sending them as a pull request. I don't expect
>>>> you to actually pull them as it is, but just to pull them on a local
>>>> branch and see what they look like.
>>>
>>> I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
>>> patches.  I am reflecting those changes in my bib sources as well.
>>> Looked sane at first glance, but yes, I need to work out how to handle
>>> the later ones with other documents...
>>
>> And the later .bib edits seem compatible with my current tools, even
>> without the script and .bst changes.  So I have applied them in tandem
>> to my .bib source and to the perfbook bibliography.  I was able to
>> find valid URLs for a few of the entries, so used them instead of
>> \nolinkurl{}, but several do appear to be quite dead.
>>
>> I also applied the alphapf.bst changes, but left inlinelinks disabled
>> for the time being.  (I am concerned about leaving authors off.)
>>
>> Thank you very much for your work on this!!!
> 
> You are welcome!
> 
> Now, the branch "bib-append-doi" might have some conflicts.
> I'll rebase it and submit a v2 of the PULL request.
> 
> inlinelinks seems to have upper limits of around 13 authors.
> I've not found any workaround of the issue...
> Can't you live with a short author field supplemented by a note of
> full authors list?

Hi Paul,

As you have switched to alphapf.bst, I investigated this issue.
It turned out that the length of author list is *not* the root cause.
By changing the length of the list, I unintentionally moved the
page boundary in Bibliography, and that resolved the build error.

The root cause of the error was an href-ed *long* title which crossed
a page boundary.

In perfbook, you can't control page breaks in Bibliography by the
nature of the project. So, I wrote a perl script to convert output of
BiBTeX on the fly.

Now, you can enable "inlinelins". Note that in appending url info to
.bib files, we may need to add more conversion patters. 

                                             Thanks, Akira

--
Akira Yokosawa (2):
  Add script to tweak output of 'alphapf' bibliography stile
  alphapf.bst: Enable 'inlinelinks'

 Makefile                    |  1 +
 alphapf.bst                 |  2 +-
 utilities/tweakhrefinbbl.pl | 37 +++++++++++++++++++++++++++++++++++++
 3 files changed, 39 insertions(+), 1 deletion(-)
 create mode 100644 utilities/tweakhrefinbbl.pl

-- 
2.7.4

> 
>                                                    Thanks, Akira
> 
>>
>> 							Thanx, Paul
>>
> [snip]
> 

[PATCH 1/2] Add script to tweak output of 'alphapf' bibliography stile

[Akira Yokosawa]

From 446d9fb1743d3750b06923e32f449469cb1364de Mon Sep 17 00:00:00 2001
From: Akira Yokosawa <akiyks@gmail.com>
Date: Sun, 25 Dec 2016 19:59:58 +0900
Subject: [PATCH 1/2] Add script to tweak output of 'alphapf' bibliography stile

"inlinelinks" in alphapf.bst caused PDFTeX to abort.
This was because href-ed long titles in Bibliography can cross
page boundaries. The combination of PDFTeX and "hyperref" package
can't handle such a situation.
You can avoid the issue by manually adjust page breaks. But in
perfbook, page boundaries in Bibliography can not be controlled
precisely.
This commit adds a script to convert \href related output of BiBTeX
on the fly. It shortens the href-ed part in title strings to the
beginning of two characters.

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
---
 Makefile                    |  1 +
 utilities/tweakhrefinbbl.pl | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100644 utilities/tweakhrefinbbl.pl

diff --git a/Makefile b/Makefile
index f0944f0..0ace9dc 100644
--- a/Makefile
+++ b/Makefile
@@ -70,6 +70,7 @@ $(PDFTARGETS): %.pdf: %.tex %.bbl

 $(PDFTARGETS:.pdf=.bbl): %.bbl: %.aux
 	bibtex $(basename $@)
+	perl utilities/tweakhrefinbbl.pl $@

 $(PDFTARGETS:.pdf=.aux): $(LATEXSOURCES) $(LATEXGENERATED) $(BIBSOURCES)
 	sh utilities/runfirstlatex.sh $(basename $@)
diff --git a/utilities/tweakhrefinbbl.pl b/utilities/tweakhrefinbbl.pl
new file mode 100644
index 0000000..6727a28
--- /dev/null
+++ b/utilities/tweakhrefinbbl.pl
@@ -0,0 +1,37 @@
+#!/usr/bin/perl -pi
+# Tweak output of BiBTeX with alphapf bibliography style
+#
+# This script does the following conversion in .bbl
+#
+#  o \newblock \href {http://URL} {Title of the site}.
+#    ->
+#    \newblock \href {http://URL} {Ti}{tle of the site}.
+#  o \newblock \href {http://URL} {{\em Book Title}}.
+#    ->
+#    \newblock \href {http://URL} {{\em {Bo}}}{\em {ok Title}}.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, you can access it online at
+# http://www.gnu.org/licenses/gpl-2.0.html.
+#
+# Copyright (C) Akira Yokosawa, 2016
+#
+# Authors: Akira Yokosawa <akiyks@gmail.com>
+
+use strict;
+use warnings;
+
+BEGIN {undef $/;}
+
+s/\\newblock \\href\s+\{([^}]*)\}\s+\{([^\\\{]{1}\S?)(.*?)\}\.$/\\newblock \\href {$1} {$2}{$3}./smg ;
+s/\\newblock \\href\s+\{([^}]*)\}\s+\{\{\\em\s+([^\\\{]{1}\S?)(.*?)\}\.$/\\newblock \\href {$1} {{\\em {$2}}}{\\em {$3}./smg ;
-- 
2.7.4

[PATCH 2/2] alphapf.bst: Enable 'inlinelinks'

[Akira Yokosawa]

From 651a65ca5cc7c0a1b14a022c8295f43f329affa3 Mon Sep 17 00:00:00 2001
From: Akira Yokosawa <akiyks@gmail.com>
Date: Sat, 5 Nov 2016 08:40:25 +0900
Subject: [PATCH 2/2] alphapf.bst: Enable 'inlinelinks'

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
---
 alphapf.bst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/alphapf.bst b/alphapf.bst
index 498e0ee..6a9ec4d 100644
--- a/alphapf.bst
+++ b/alphapf.bst
@@ -69,7 +69,7 @@ FUNCTION {init.urlbst.variables}
   #1 'adddoiresolver :=     % 0=no DOI resolver; 1=include it
   #1 'addpubmedresolver :=     % 0=no PUBMED resolver; 1=include it
   #2 'hrefform :=           % 0=no crossrefs; 1=hypertex xrefs; 2=hyperref refs
-  #0 'inlinelinks :=        % 0=URLs explicit; 1=URLs attached to titles
+  #1 'inlinelinks :=        % 0=URLs explicit; 1=URLs attached to titles

   % String constants, which you _might_ want to tweak.
   "URL: " 'urlintro := % prefix before URL; typically "Available from:" or "URL":
-- 
2.7.4

Re: [PATCH 0/2] Add script to convert BiBTeX output (was Re: [RFC PULL] Bibliography URL cleanup)

[Paul E. McKenney]

On Sun, Dec 25, 2016 at 08:42:18PM +0900, Akira Yokosawa wrote:
> On 2016/12/24 16:36:32 +0900, Akira Yokosawa wrote:
> > On 2016/12/24 23:05:05 -0800, Paul E. McKenney wrote:
> >> On Sun, Nov 06, 2016 at 03:47:19PM -0800, Paul E. McKenney wrote:
> >>> On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
> >>>> Hi Paul,
> >>>>
> >>>> On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
> >>>>> On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
> >>>> [snip]
> >>>>>> So, these bib files are an library collected for nearly three decades!!!
> >>>>>> They are invaluable as they are, and I'd appreciate your decision to
> >>>>>> make them public.
> >>>>>
> >>>>> Unfortunately, many of the comments on the early entries reflect my
> >>>>> relative youth and impetuosity, so unless or until I get time to edit
> >>>>> the whole mess so as to avoid offending any number of authors (to say
> >>>>> nothing of their disciples!), I must keep the originals private.
> >>>>
> >>>> I see. I misunderstood the circumstances. So you made only a part of your
> >>>> bib files public.
> >>>>
> >>>>>
> >>>>>> There are two issues in urls in the bib files.
> >>>>>> One is the inconsistency of format discussed here.
> >>>>>> The other is the dead links. There are quite a few urls that end up in
> >>>>>> "not found" now. Maintaining urls would require a great deal of work itself...
> >>>>>>
> >>>>>> To make the format consistent, a script would work. But before beginning
> >>>>>> implementation, we need to clarify what the script would do.
> >>>>>> So I'll make some sample replacement patches to confirm your preference.
> >>>>>
> >>>>> Sounds good, and I look forward to seeing them!
> >>>>
> >>>> I said to make "some sample replacement patches", but it became quite
> >>>> intensive changes. So I'm sending them as a pull request. I don't expect
> >>>> you to actually pull them as it is, but just to pull them on a local
> >>>> branch and see what they look like.
> >>>
> >>> I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
> >>> patches.  I am reflecting those changes in my bib sources as well.
> >>> Looked sane at first glance, but yes, I need to work out how to handle
> >>> the later ones with other documents...
> >>
> >> And the later .bib edits seem compatible with my current tools, even
> >> without the script and .bst changes.  So I have applied them in tandem
> >> to my .bib source and to the perfbook bibliography.  I was able to
> >> find valid URLs for a few of the entries, so used them instead of
> >> \nolinkurl{}, but several do appear to be quite dead.
> >>
> >> I also applied the alphapf.bst changes, but left inlinelinks disabled
> >> for the time being.  (I am concerned about leaving authors off.)
> >>
> >> Thank you very much for your work on this!!!
> > 
> > You are welcome!
> > 
> > Now, the branch "bib-append-doi" might have some conflicts.
> > I'll rebase it and submit a v2 of the PULL request.
> > 
> > inlinelinks seems to have upper limits of around 13 authors.
> > I've not found any workaround of the issue...
> > Can't you live with a short author field supplemented by a note of
> > full authors list?
> 
> Hi Paul,
> 
> As you have switched to alphapf.bst, I investigated this issue.
> It turned out that the length of author list is *not* the root cause.
> By changing the length of the list, I unintentionally moved the
> page boundary in Bibliography, and that resolved the build error.
> 
> The root cause of the error was an href-ed *long* title which crossed
> a page boundary.

Thank you for tracking this down!  This is a title as in the title
of a citation, or as in the title of an href?  (My guess is the latter,
but it has been about 20 years since I last used perl.)

> In perfbook, you can't control page breaks in Bibliography by the
> nature of the project. So, I wrote a perl script to convert output of
> BiBTeX on the fly.

Indeed, and this presumably means that the current project is
randomly vulnerable to this bug.  Are we the only ones to see it?
If not, is a fix on the way?

> Now, you can enable "inlinelins". Note that in appending url info to
> .bib files, we may need to add more conversion patters. 

Actually, my current thought is to revert the alpha.bst commits
pending a fix.  Though I don't see the need to be too quick about
it, given that it seems to work for both 1c and 2c at the moment.

Or am I missing some reason why this cannot be fixed?

							Thanx, Paul

>                                              Thanks, Akira
> 
> --
> Akira Yokosawa (2):
>   Add script to tweak output of 'alphapf' bibliography stile
>   alphapf.bst: Enable 'inlinelinks'
> 
>  Makefile                    |  1 +
>  alphapf.bst                 |  2 +-
>  utilities/tweakhrefinbbl.pl | 37 +++++++++++++++++++++++++++++++++++++
>  3 files changed, 39 insertions(+), 1 deletion(-)
>  create mode 100644 utilities/tweakhrefinbbl.pl
> 
> -- 
> 2.7.4
> 
> > 
> >                                                    Thanks, Akira
> > 
> >>
> >> 							Thanx, Paul
> >>
> > [snip]
> > 
> 

Re: [PATCH 0/2] Add script to convert BiBTeX output (was Re: [RFC PULL] Bibliography URL cleanup)

[Akira Yokosawa]

On 2016/12/25 08:15:09 -0800, Paul E. McKenney wrote:
> On Sun, Dec 25, 2016 at 08:42:18PM +0900, Akira Yokosawa wrote:
>> On 2016/12/24 16:36:32 +0900, Akira Yokosawa wrote:
>>> On 2016/12/24 23:05:05 -0800, Paul E. McKenney wrote:
>>>> On Sun, Nov 06, 2016 at 03:47:19PM -0800, Paul E. McKenney wrote:
>>>>> On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
>>>>>> Hi Paul,
>>>>>>
>>>>>> On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
>>>>>>> On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
>>>>>> [snip]
>>>>>>>> So, these bib files are an library collected for nearly three decades!!!
>>>>>>>> They are invaluable as they are, and I'd appreciate your decision to
>>>>>>>> make them public.
>>>>>>>
>>>>>>> Unfortunately, many of the comments on the early entries reflect my
>>>>>>> relative youth and impetuosity, so unless or until I get time to edit
>>>>>>> the whole mess so as to avoid offending any number of authors (to say
>>>>>>> nothing of their disciples!), I must keep the originals private.
>>>>>>
>>>>>> I see. I misunderstood the circumstances. So you made only a part of your
>>>>>> bib files public.
>>>>>>
>>>>>>>
>>>>>>>> There are two issues in urls in the bib files.
>>>>>>>> One is the inconsistency of format discussed here.
>>>>>>>> The other is the dead links. There are quite a few urls that end up in
>>>>>>>> "not found" now. Maintaining urls would require a great deal of work itself...
>>>>>>>>
>>>>>>>> To make the format consistent, a script would work. But before beginning
>>>>>>>> implementation, we need to clarify what the script would do.
>>>>>>>> So I'll make some sample replacement patches to confirm your preference.
>>>>>>>
>>>>>>> Sounds good, and I look forward to seeing them!
>>>>>>
>>>>>> I said to make "some sample replacement patches", but it became quite
>>>>>> intensive changes. So I'm sending them as a pull request. I don't expect
>>>>>> you to actually pull them as it is, but just to pull them on a local
>>>>>> branch and see what they look like.
>>>>>
>>>>> I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
>>>>> patches.  I am reflecting those changes in my bib sources as well.
>>>>> Looked sane at first glance, but yes, I need to work out how to handle
>>>>> the later ones with other documents...
>>>>
>>>> And the later .bib edits seem compatible with my current tools, even
>>>> without the script and .bst changes.  So I have applied them in tandem
>>>> to my .bib source and to the perfbook bibliography.  I was able to
>>>> find valid URLs for a few of the entries, so used them instead of
>>>> \nolinkurl{}, but several do appear to be quite dead.
>>>>
>>>> I also applied the alphapf.bst changes, but left inlinelinks disabled
>>>> for the time being.  (I am concerned about leaving authors off.)
>>>>
>>>> Thank you very much for your work on this!!!
>>>
>>> You are welcome!
>>>
>>> Now, the branch "bib-append-doi" might have some conflicts.
>>> I'll rebase it and submit a v2 of the PULL request.
>>>
>>> inlinelinks seems to have upper limits of around 13 authors.
>>> I've not found any workaround of the issue...
>>> Can't you live with a short author field supplemented by a note of
>>> full authors list?
>>
>> Hi Paul,
>>
>> As you have switched to alphapf.bst, I investigated this issue.
>> It turned out that the length of author list is *not* the root cause.
>> By changing the length of the list, I unintentionally moved the
>> page boundary in Bibliography, and that resolved the build error.
>>
>> The root cause of the error was an href-ed *long* title which crossed
>> a page boundary.
> 
> Thank you for tracking this down!  This is a title as in the title
> of a citation, or as in the title of an href?  (My guess is the latter,
> but it has been about 20 years since I last used perl.)

Yes, the latter.

> 
>> In perfbook, you can't control page breaks in Bibliography by the
>> nature of the project. So, I wrote a perl script to convert output of
>> BiBTeX on the fly.
> 
> Indeed, and this presumably means that the current project is
> randomly vulnerable to this bug.  Are we the only ones to see it?
> If not, is a fix on the way?

If we were the only ones, I could have never found the root cause :-)
I did a search of the error message:
"! pdfTeX error (ext4): \pdfendlink ended up in different nesting
level than \pdfstartlink."

It brought me to a page at:

http://tex.stackexchange.com/questions/1522/pdfendlink-ended-up-in-different-nesting-level-than-pdfstartlink

This issue was opened 6 years ago. A recent comment (of a workaround) there
was given by Heiko Oberdiek who is one of the current maintainers of hyperref
package.

There is another page at

http://tex.stackexchange.com/questions/249579/pdfendlink-ended-up-in-different-nesting-level-than-pdfstartlink-error-with

It says that two column layout is related to the issue.

You can also find the following description in Limitations section of hyperref's README:

  Links across pages
  In general they have problems:
  * Some driver doesn’t support them at all (see above).
  * The driver allows it, but the link result might include
  the footer and/or header, or an error message can
  occur sometimes.

So, I can say that this issue is well recognized by the maintainers.

> 
>> Now, you can enable "inlinelins". Note that in appending url info to
>> .bib files, we may need to add more conversion patters. 
> 
> Actually, my current thought is to revert the alpha.bst commits
> pending a fix.  Though I don't see the need to be too quick about
> it, given that it seems to work for both 1c and 2c at the moment.
> 
> Or am I missing some reason why this cannot be fixed?

It seems unlikely to be fixed soon.

I remember tweaking book.cls to fix indentation behavior in
two column layout. My impression is that two column layout is not
as reliable as one column one...

One idea is to use alphapf.bst only for 1c with inlinelinks enabled. 

Thoughts?

And I have no objection to reverting commit ca109cf61bbf for now.

                                               Thanks, Akira

> 							Thanx, Paul
> 
>>                                              Thanks, Akira
>>
>> --
>> Akira Yokosawa (2):
>>   Add script to tweak output of 'alphapf' bibliography stile
>>   alphapf.bst: Enable 'inlinelinks'
>>
>>  Makefile                    |  1 +
>>  alphapf.bst                 |  2 +-
>>  utilities/tweakhrefinbbl.pl | 37 +++++++++++++++++++++++++++++++++++++
>>  3 files changed, 39 insertions(+), 1 deletion(-)
>>  create mode 100644 utilities/tweakhrefinbbl.pl
>>
>> -- 
>> 2.7.4
>>
>>>
>>>                                                    Thanks, Akira
>>>
>>>>
>>>> 							Thanx, Paul
>>>>
>>> [snip]
>>>
>>
> 
> 

Re: [PATCH 0/2] Add script to convert BiBTeX output (was Re: [RFC PULL] Bibliography URL cleanup)

[Paul E. McKenney]

On Mon, Dec 26, 2016 at 11:52:40PM +0900, Akira Yokosawa wrote:
> On 2016/12/25 08:15:09 -0800, Paul E. McKenney wrote:
> > On Sun, Dec 25, 2016 at 08:42:18PM +0900, Akira Yokosawa wrote:
> >> On 2016/12/24 16:36:32 +0900, Akira Yokosawa wrote:
> >>> On 2016/12/24 23:05:05 -0800, Paul E. McKenney wrote:
> >>>> On Sun, Nov 06, 2016 at 03:47:19PM -0800, Paul E. McKenney wrote:
> >>>>> On Sat, Nov 05, 2016 at 05:16:19PM +0900, Akira Yokosawa wrote:
> >>>>>> Hi Paul,
> >>>>>>
> >>>>>> On 2016/10/28, 11:30:38 -0700, Paul E. McKenney wrote:
> >>>>>>> On Fri, Oct 28, 2016 at 07:45:16AM +0900, Akira Yokosawa wrote:
> >>>>>> [snip]
> >>>>>>>> So, these bib files are an library collected for nearly three decades!!!
> >>>>>>>> They are invaluable as they are, and I'd appreciate your decision to
> >>>>>>>> make them public.
> >>>>>>>
> >>>>>>> Unfortunately, many of the comments on the early entries reflect my
> >>>>>>> relative youth and impetuosity, so unless or until I get time to edit
> >>>>>>> the whole mess so as to avoid offending any number of authors (to say
> >>>>>>> nothing of their disciples!), I must keep the originals private.
> >>>>>>
> >>>>>> I see. I misunderstood the circumstances. So you made only a part of your
> >>>>>> bib files public.
> >>>>>>
> >>>>>>>
> >>>>>>>> There are two issues in urls in the bib files.
> >>>>>>>> One is the inconsistency of format discussed here.
> >>>>>>>> The other is the dead links. There are quite a few urls that end up in
> >>>>>>>> "not found" now. Maintaining urls would require a great deal of work itself...
> >>>>>>>>
> >>>>>>>> To make the format consistent, a script would work. But before beginning
> >>>>>>>> implementation, we need to clarify what the script would do.
> >>>>>>>> So I'll make some sample replacement patches to confirm your preference.
> >>>>>>>
> >>>>>>> Sounds good, and I look forward to seeing them!
> >>>>>>
> >>>>>> I said to make "some sample replacement patches", but it became quite
> >>>>>> intensive changes. So I'm sending them as a pull request. I don't expect
> >>>>>> you to actually pull them as it is, but just to pull them on a local
> >>>>>> branch and see what they look like.
> >>>>>
> >>>>> I have pulled thme into akiyks.2016.11.05a, and pushed the first nine
> >>>>> patches.  I am reflecting those changes in my bib sources as well.
> >>>>> Looked sane at first glance, but yes, I need to work out how to handle
> >>>>> the later ones with other documents...
> >>>>
> >>>> And the later .bib edits seem compatible with my current tools, even
> >>>> without the script and .bst changes.  So I have applied them in tandem
> >>>> to my .bib source and to the perfbook bibliography.  I was able to
> >>>> find valid URLs for a few of the entries, so used them instead of
> >>>> \nolinkurl{}, but several do appear to be quite dead.
> >>>>
> >>>> I also applied the alphapf.bst changes, but left inlinelinks disabled
> >>>> for the time being.  (I am concerned about leaving authors off.)
> >>>>
> >>>> Thank you very much for your work on this!!!
> >>>
> >>> You are welcome!
> >>>
> >>> Now, the branch "bib-append-doi" might have some conflicts.
> >>> I'll rebase it and submit a v2 of the PULL request.
> >>>
> >>> inlinelinks seems to have upper limits of around 13 authors.
> >>> I've not found any workaround of the issue...
> >>> Can't you live with a short author field supplemented by a note of
> >>> full authors list?
> >>
> >> Hi Paul,
> >>
> >> As you have switched to alphapf.bst, I investigated this issue.
> >> It turned out that the length of author list is *not* the root cause.
> >> By changing the length of the list, I unintentionally moved the
> >> page boundary in Bibliography, and that resolved the build error.
> >>
> >> The root cause of the error was an href-ed *long* title which crossed
> >> a page boundary.
> > 
> > Thank you for tracking this down!  This is a title as in the title
> > of a citation, or as in the title of an href?  (My guess is the latter,
> > but it has been about 20 years since I last used perl.)
> 
> Yes, the latter.
> 
> > 
> >> In perfbook, you can't control page breaks in Bibliography by the
> >> nature of the project. So, I wrote a perl script to convert output of
> >> BiBTeX on the fly.
> > 
> > Indeed, and this presumably means that the current project is
> > randomly vulnerable to this bug.  Are we the only ones to see it?
> > If not, is a fix on the way?
> 
> If we were the only ones, I could have never found the root cause :-)
> I did a search of the error message:
> "! pdfTeX error (ext4): \pdfendlink ended up in different nesting
> level than \pdfstartlink."
> 
> It brought me to a page at:
> 
> http://tex.stackexchange.com/questions/1522/pdfendlink-ended-up-in-different-nesting-level-than-pdfstartlink
> 
> This issue was opened 6 years ago. A recent comment (of a workaround) there
> was given by Heiko Oberdiek who is one of the current maintainers of hyperref
> package.
> 
> There is another page at
> 
> http://tex.stackexchange.com/questions/249579/pdfendlink-ended-up-in-different-nesting-level-than-pdfstartlink-error-with
> 
> It says that two column layout is related to the issue.
> 
> You can also find the following description in Limitations section of hyperref's README:
> 
>   Links across pages
>   In general they have problems:
>   * Some driver doesn’t support them at all (see above).
>   * The driver allows it, but the link result might include
>   the footer and/or header, or an error message can
>   occur sometimes.
> 
> So, I can say that this issue is well recognized by the maintainers.

And it does sound like you extensively researched this issue!  Thank
for doing so!

> >> Now, you can enable "inlinelins". Note that in appending url info to
> >> .bib files, we may need to add more conversion patters. 
> > 
> > Actually, my current thought is to revert the alpha.bst commits
> > pending a fix.  Though I don't see the need to be too quick about
> > it, given that it seems to work for both 1c and 2c at the moment.
> > 
> > Or am I missing some reason why this cannot be fixed?
> 
> It seems unlikely to be fixed soon.
> 
> I remember tweaking book.cls to fix indentation behavior in
> two column layout. My impression is that two column layout is not
> as reliable as one column one...
> 
> One idea is to use alphapf.bst only for 1c with inlinelinks enabled. 
> 
> Thoughts?

I am good with enabling alphaph.bst inlinelinks only for 1c!

							Thanx, Paul

> And I have no objection to reverting commit ca109cf61bbf for now.
> 
>                                                Thanks, Akira
> 
> > 							Thanx, Paul
> > 
> >>                                              Thanks, Akira
> >>
> >> --
> >> Akira Yokosawa (2):
> >>   Add script to tweak output of 'alphapf' bibliography stile
> >>   alphapf.bst: Enable 'inlinelinks'
> >>
> >>  Makefile                    |  1 +
> >>  alphapf.bst                 |  2 +-
> >>  utilities/tweakhrefinbbl.pl | 37 +++++++++++++++++++++++++++++++++++++
> >>  3 files changed, 39 insertions(+), 1 deletion(-)
> >>  create mode 100644 utilities/tweakhrefinbbl.pl
> >>
> >> -- 
> >> 2.7.4
> >>
> >>>
> >>>                                                    Thanks, Akira
> >>>
> >>>>
> >>>> 							Thanx, Paul
> >>>>
> >>> [snip]
> >>>
> >>
> > 
> > 
>