Re: marching through all physical memory in software

Re: marching through all physical memory in software

[Eric W. Biederman]

Doug Thompson <norsk5@yahoo.com> writes:

> Nigel Cunningham <ncunningham-lkml@crca.org.au> wrote:
>
>     Hi again.
>
>     On Fri, 2009-01-30 at 10:13 +0100, Pavel Machek wrote:
>     > > Hi.
>     > >
>     > > On Wed, 2009-01-28 at 20:38 +0100, Pavel Machek wrote:
>     > > > You can do the scrubbing today by echo reboot > /sys/power/disk; echo
>     > > > disk > /sys/power/state :-)... or using uswsusp APIs.
>     > >
>     > > That won't work. The RAM retains its contents across a reboot, and even
>     > > for a little while after powering off.
>     >
>     > Yes, and the original goal was to rewrite all the memory with same
>     > contents so that parity errors don't accumulate. SO scrubbing here !=
>     > trying to clear it.
>
>     Sorry - I think I missed something.
>
>     AFAICS, hibernating is going to be a noop as far as doing anything to
>     memory that's not touched by the process of hibernating goes. It won't
>     clear it or scrub it or anything else.

A background software scrubber simply has the job of rewritting memory
to it's current content so that the data and the ecc check bits are
guaranteed to be in sync keeping correctable ecc errors caused by
environmental factors from accumulating.

Pavel's original comment was that the hibernation code has to walk all
of memory to save it to disk so it would be a good place to look to
figure out how to walk all of memory.  And incidentally hibernation
would serve as a crud way of rewritting all of memory.


> Even if hibernating worked, it does not touch the issue of scrubbing memory
> that doesn't have hardware support AND the requirement of thousands of nodes in
> cluster who MUST remain operational for days on end.

But it may still serve as an example of how to walk through all of memory.

> Sicortex's MIPS based system fits that exactly.  When I did their EDAC driver
> they wanted to have a software scrubber at a NICE run level to scan memory and
> do this operation without shutting down the system.
>
> We never got to it, but it would be a nice for some to have a background
> software scrubber.  But I would need help from the memory guys on a proper
> interface.
>
> The goal would be have a "loose" target of attempting to all most memory if not
> all.  Sometime of iteration over the memory set.

Thinking about it.  We only care about memory the kernel is using so the memory
maps the BIOS supplies the kernel should be sufficient.  We have weird corner
cases like ACPI but not handling those in the first pass and getting
something working should be fine.

There are other silly things such as wanting to only scrub memory on it's native
NUMA node (if possible) for both performance and scalability.

Eric

Re: marching through all physical memory in software

[Tim Small]

Eric W. Biederman wrote:
> A background software scrubber simply has the job of rewritting memory
> to it's current content so that the data and the ecc check bits are
> guaranteed to be in sync

Don't you just need to READ memory?  The memory controller hardware 
takes care of the rest in the vast majority of cases.

You only need to rewrite RAM if a correctable error occurs, and the 
chipset doesn't support automatic write-back of the corrected value (a 
different problem altogether...).  The actual memory bits themselves are 
refreshed by the hardware quite frequently (max of every 64ms for DDR2, 
I believe)...

Cheers,

Tim.

Re: marching through all physical memory in software

[Eric W. Biederman]

Tim Small <tim@buttersideup.com> writes:

> Eric W. Biederman wrote:
>> A background software scrubber simply has the job of rewritting memory
>> to it's current content so that the data and the ecc check bits are
>> guaranteed to be in sync
>
> Don't you just need to READ memory?  The memory controller hardware takes care
> of the rest in the vast majority of cases.
>
> You only need to rewrite RAM if a correctable error occurs, and the chipset
> doesn't support automatic write-back of the corrected value (a different problem
> altogether...).  The actual memory bits themselves are refreshed by the hardware
> quite frequently (max of every 64ms for DDR2, I believe)...

At the point we are talking about software scrubbing it makes sense to assume
a least common denominator memory controller, one that does not do automatic
write-back of the corrected value, as all of the recent memory controllers
do scrubbing in hardware.

Once you handle the stupidest hardware all other cases are just software optimizations
on that, and we already have the tricky code that does a read-modify-write without
changing the contents of memory, so guarantees everything it touches will be written
back.

Eric

Re: marching through all physical memory in software

[Tim Small]

Eric W. Biederman wrote:
> At the point we are talking about software scrubbing it makes sense to assume
> a least common denominator memory controller, one that does not do automatic
> write-back of the corrected value, as all of the recent memory controllers
> do scrubbing in hardware.
>   

I was just trying to clarify the distinction between the two processes 
which have similar names, but aren't (IMO) actually that similar:

"Software Scrubbing"

Triggering a read, and subsequent rewrite of a particular RAM location 
which has suffered a correctable ECC error(s) i.e. hardware detects an 
error, then the OS takes care of the rewrite to "scrub" the error in the 
case that the hardware doesn't handle this automatically.

This should be a very-occasional error-path process, and performance is 
probably not critical..


"Background Scrubbing"

. This is a poor name, IMO (scrub infers some kind of write to me), 
which applies to a process whereby you ensure that the ECC check-bits 
are verified periodically for the whole of physical RAM, so that single 
bit errors in a given ECC block don't accumulate and turn into 
uncorrectable errors.  It may also lead to improved data collection for 
some failure modes.  Again, many memory controllers implement this 
feature in hardware, so we shouldn't do it twice where this is supported.

There is (AFAIK) no need to do any writes here, and in fact doing so is 
only likely to hurt performance, I think....  The design which springs 
to mind is of a background thread which (possibly at idle priority) 
reads RAM at a user-configurable rate (e.g. consume a max of n% of 
memory bandwidth, or read  all of RAM at least once every x minutes).  
Possible design issues:

. There will be some trade off between reducing impact on the system as 
a whole, and making firm guarantees about how often memory is checked.  
Difficult to know what the default would be, but probably 
no-firm-guarantee of minimum time (idle processing only) is likely to 
cause least problems for most users.
. An eye will need to be kept on the impact that this reading has on the 
performance of the rest of the system (e.g. cache pollution, and NUMA, 
as you previously mentioned), but my gut feeling is that for the 
majority of systems it shouldn't be significant.  If practical 
mechanisms are available on some CPUs to read RAM without populating the 
CPU cache, we should use those (but I've no idea if they exist or not).

Perhaps a good default would be to benchmark memory read bandwidth when 
the feature is turned on, and then operate at (e.g.) 0.5% of that bandwidth.


Cheers,

Tim.

Re: marching through all physical memory in software

[Henrique de Moraes Holschuh]

On Sat, 31 Jan 2009, Tim Small wrote:
> Eric W. Biederman wrote:
> > At the point we are talking about software scrubbing it makes sense to assume
> > a least common denominator memory controller, one that does not do automatic
> > write-back of the corrected value, as all of the recent memory controllers
> > do scrubbing in hardware.
> >   
> 
> I was just trying to clarify the distinction between the two processes 
> which have similar names, but aren't (IMO) actually that similar:
> 
> "Software Scrubbing"
> 
> Triggering a read, and subsequent rewrite of a particular RAM location 
> which has suffered a correctable ECC error(s) i.e. hardware detects an 
> error, then the OS takes care of the rewrite to "scrub" the error in the 
> case that the hardware doesn't handle this automatically.
> 
> This should be a very-occasional error-path process, and performance is 
> probably not critical..
> 
> 
> "Background Scrubbing"
> 
> . This is a poor name, IMO (scrub infers some kind of write to me), 
> which applies to a process whereby you ensure that the ECC check-bits 
> are verified periodically for the whole of physical RAM, so that single 
> bit errors in a given ECC block don't accumulate and turn into 
> uncorrectable errors.  It may also lead to improved data collection for 
> some failure modes.  Again, many memory controllers implement this 
> feature in hardware, so we shouldn't do it twice where this is supported.

It is implined in the background scrubbing, that if a background scrub
page read causes an ECC correctable error to be flagged, the normal
"fix through scrub" behaviour of the memory controller will be
triggered (possibly, the software scrubbing described above).

And if an uncorretable error is detected during the scrub, we have to
do something about it as well.  And that won't be that easy: locate
whatever process is using that page, and so something smart to it...
or do some emergency evasive actions if it is one of the kernel's data
scructures, etc.

So, as you said, "background scrubbing" and "software scrubbing" really are
very different things, and one has to expect that background scrubbing will
eventually trigger software scrubbing, major system emergency handling
(uncorrectable errors in kernel memory) or minor system emergency
handling (uncorrectable errors in process memory).

> There is (AFAIK) no need to do any writes here, and in fact doing so is 

One might want the possibility of doing inconditional writes, because
it helps with memory bitrot on crappy hardware where the refresh
cycles aren't enough to avoid bitrot.  But you definately won't want
it most of the time.

You can also implement software-based ECC using a background scrubber
and setting aside pages to store the ECC information.  Now, THAT is
probably not worth bothering with due to the performance impact, but
who knows...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: marching through all physical memory in software

[Pavel Machek]

Hi!

> And if an uncorretable error is detected during the scrub, we have to
> do something about it as well.  And that won't be that easy: locate
> whatever process is using that page, and so something smart to it...
> or do some emergency evasive actions if it is one of the kernel's data
> scructures, etc.
> 
> So, as you said, "background scrubbing" and "software scrubbing" really are
> very different things, and one has to expect that background scrubbing will
> eventually trigger software scrubbing, major system emergency handling
> (uncorrectable errors in kernel memory) or minor system emergency
> handling (uncorrectable errors in process memory).
> 
> > There is (AFAIK) no need to do any writes here, and in fact doing so is 
> 
> One might want the possibility of doing inconditional writes, because
> it helps with memory bitrot on crappy hardware where the refresh
> cycles aren't enough to avoid bitrot.  But you definately won't want
> it most of the time.
> 
> You can also implement software-based ECC using a background scrubber
> and setting aside pages to store the ECC information.  Now, THAT is
> probably not worth bothering with due to the performance impact, but
> who knows...

Actually, that would be quite cool. a) I suspect memory in  my zaurus
bitrots and b) bitroting memory over s2ram is apprently quite common.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: marching through all physical memory in software

[Henrique de Moraes Holschuh]

On Sat, 31 Jan 2009, Pavel Machek wrote:
> > You can also implement software-based ECC using a background scrubber
> > and setting aside pages to store the ECC information.  Now, THAT is
> > probably not worth bothering with due to the performance impact, but
> > who knows...
> 
> Actually, that would be quite cool. a) I suspect memory in  my zaurus
> bitrots and b) bitroting memory over s2ram is apprently quite common.

Well, software-based ECC for s2ram (calculate right before s2ram,
check-and-fix right after waking up) is certainly doable and a LOT
easier than my crazy idea of sofware-based generic ECC (which requires
some sort of trick to detect pages that were written to, so that you
can update their ECC information)...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: marching through all physical memory in software

[Nigel Cunningham]

Hi.

On Fri, 2009-01-30 at 11:32 -0800, Eric W. Biederman wrote:
> Doug Thompson <norsk5@yahoo.com> writes:
> 
> > Nigel Cunningham <ncunningham-lkml@crca.org.au> wrote:
> >
> >     Hi again.
> >
> >     On Fri, 2009-01-30 at 10:13 +0100, Pavel Machek wrote:
> >     > > Hi.
> >     > >
> >     > > On Wed, 2009-01-28 at 20:38 +0100, Pavel Machek wrote:
> >     > > > You can do the scrubbing today by echo reboot > /sys/power/disk; echo
> >     > > > disk > /sys/power/state :-)... or using uswsusp APIs.
> >     > >
> >     > > That won't work. The RAM retains its contents across a reboot, and even
> >     > > for a little while after powering off.
> >     >
> >     > Yes, and the original goal was to rewrite all the memory with same
> >     > contents so that parity errors don't accumulate. SO scrubbing here !=
> >     > trying to clear it.
> >
> >     Sorry - I think I missed something.
> >
> >     AFAICS, hibernating is going to be a noop as far as doing anything to
> >     memory that's not touched by the process of hibernating goes. It won't
> >     clear it or scrub it or anything else.
> 
> A background software scrubber simply has the job of rewritting memory
> to it's current content so that the data and the ecc check bits are
> guaranteed to be in sync keeping correctable ecc errors caused by
> environmental factors from accumulating.
> 
> Pavel's original comment was that the hibernation code has to walk all
> of memory to save it to disk so it would be a good place to look to
> figure out how to walk all of memory.  And incidentally hibernation
> would serve as a crud way of rewritting all of memory.

Thanks. Now I get it :)

Nigel

Re: marching through all physical memory in software

[Chris Friesen]

Eric W. Biederman wrote:

> Thinking about it.  We only care about memory the kernel is using so the memory
> maps the BIOS supplies the kernel should be sufficient.  We have weird corner
> cases like ACPI but not handling those in the first pass and getting
> something working should be fine.

Agreed.

The next question is who handles the conversion of the various different 
arch-specific BIOS mappings to a standard format that we can feed to the 
background "scrub" code.  Is this something that belongs in the edac 
memory controller code, or would it live in /arch/foo somewhere?

Chris

Re: marching through all physical memory in software

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

On Mon, 02 Feb 2009 12:29:45 CST, Chris Friesen said:
> The next question is who handles the conversion of the various different 
> arch-specific BIOS mappings to a standard format that we can feed to the 
> background "scrub" code.  Is this something that belongs in the edac 
> memory controller code, or would it live in /arch/foo somewhere?

If it's intended to be something basically stand-alone that doesn't require
an actual EDAC chipset, it should probably live elsewhere.  Otherwise, you get
into the case of people who don't enable it because they "know" their hardware
doesn't have an EDAC ability, even if they *could* benefit from the function.

On the other hand, if it's an EDAC-only thing, maybe under drivers/edac/$ARCH?


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: marching through all physical memory in software

[Chris Friesen]

Valdis.Kletnieks@vt.edu wrote:
> On Mon, 02 Feb 2009 12:29:45 CST, Chris Friesen said:
> 
>>The next question is who handles the conversion of the various different 
>>arch-specific BIOS mappings to a standard format that we can feed to the 
>>background "scrub" code.  Is this something that belongs in the edac 
>>memory controller code, or would it live in /arch/foo somewhere?
> 
> 
> If it's intended to be something basically stand-alone that doesn't require
> an actual EDAC chipset, it should probably live elsewhere.  Otherwise, you get
> into the case of people who don't enable it because they "know" their hardware
> doesn't have an EDAC ability, even if they *could* benefit from the function.
> 
> On the other hand, if it's an EDAC-only thing, maybe under drivers/edac/$ARCH?

I don't see anything in the name of EDAC that implies hardware only...a 
software memory scrub could be considered "error detection and 
correction".  Might have to update the config help text though.

Chris

Re: marching through all physical memory in software

[Pavel Machek]

Hi!

>>> The next question is who handles the conversion of the various 
>>> different arch-specific BIOS mappings to a standard format that we 
>>> can feed to the background "scrub" code.  Is this something that 
>>> belongs in the edac memory controller code, or would it live in 
>>> /arch/foo somewhere?
>>
>>
>> If it's intended to be something basically stand-alone that doesn't require
>> an actual EDAC chipset, it should probably live elsewhere.  Otherwise, you get
>> into the case of people who don't enable it because they "know" their hardware
>> doesn't have an EDAC ability, even if they *could* benefit from the function.
>>
>> On the other hand, if it's an EDAC-only thing, maybe under drivers/edac/$ARCH?
>
> I don't see anything in the name of EDAC that implies hardware only...a  
> software memory scrub could be considered "error detection and  
> correction".  Might have to update the config help text though.

Software memory scrub would no longer be a "driver" :-). So it should
go into kernel/scrub or mm/scrub or maybe mm/edac or something.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: marching through all physical memory in software

[Chris Friesen]

Pavel Machek wrote:

>>I don't see anything in the name of EDAC that implies hardware only...a  
>>software memory scrub could be considered "error detection and  
>>correction".  Might have to update the config help text though.
> 
> 
> Software memory scrub would no longer be a "driver" :-). So it should
> go into kernel/scrub or mm/scrub or maybe mm/edac or something.

True enough.  In that case, something under "mm" makes more sense to me 
than something under "kernel".

Chris

Re: marching through all physical memory in software

[Dave Jiang]

There may be generic code, but the actual scrubbing can be architecture 
specific. You have to atomically read and write back. And each arch has 
different way of handling that. See arch/X/include/asm/edac.h

Pavel Machek wrote:
> Software memory scrub would no longer be a "driver" :-). So it should
> go into kernel/scrub or mm/scrub or maybe mm/edac or something.
> 

marching through all physical memory in software

[Chris Friesen]

Someone is asking me about the feasability of "scrubbing" system memory 
by accessing each page and handling the ECC faults.

The range of PAGE_OFFSET to "high_memory" should get me all of the 
kernel memory area, but what about all the memory set aside for 
userspace (which may not be contiguous)?  Is there any straightforward 
way to march through this memory?

I suppose I'm looking for something like walk_page_range(), but for 
physical memory rather than virtual.

Thanks,

Chris

Re: marching through all physical memory in software

[Arjan van de Ven]

On Mon, 26 Jan 2009 09:38:13 -0600
"Chris Friesen" <cfriesen@nortel.com> wrote:

> Someone is asking me about the feasability of "scrubbing" system
> memory by accessing each page and handling the ECC faults.
> 

Hi,

I would suggest that you look at the "edac" subsystem, which tries to
do exactly this....



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: marching through all physical memory in software

[Chris Friesen]

Arjan van de Ven wrote:
> On Mon, 26 Jan 2009 09:38:13 -0600
> "Chris Friesen" <cfriesen@nortel.com> wrote:
> 
>> Someone is asking me about the feasability of "scrubbing" system
>> memory by accessing each page and handling the ECC faults.
>>
> 
> Hi,
> 
> I would suggest that you look at the "edac" subsystem, which tries to
> do exactly this....

Looking at the current -git code, there appears to be an option for 
memory controllers to do this (the set_sdram_scrub_rate() routine), but 
there don't appear to be any controllers that can actually do it.

edac appears to currently be able to scrub the specific page where the 
fault occurred.  This is a useful building block, but doesn't provide 
the ability to march through all of physical memory.

Chris

Re: marching through all physical memory in software

[Eric W. Biederman]

"Chris Friesen" <cfriesen@nortel.com> writes:

> Arjan van de Ven wrote:
>> On Mon, 26 Jan 2009 09:38:13 -0600
>> "Chris Friesen" <cfriesen@nortel.com> wrote:
>>
>>> Someone is asking me about the feasability of "scrubbing" system
>>> memory by accessing each page and handling the ECC faults.
>>>
>>
>> Hi,
>>
>> I would suggest that you look at the "edac" subsystem, which tries to
>> do exactly this....


> edac appears to currently be able to scrub the specific page where the fault
> occurred.  This is a useful building block, but doesn't provide the ability to
> march through all of physical memory.

Well that is the tricky part.  The rest is simply finding which physical
addresses are valid.  Either by querying the memory controller or looking
at the range the BIOS gave us.

That part should not be too hard.  I think it simply has not been implemented
yet as most ECC chipsets implement this in hardware today.

Eric

Re: marching through all physical memory in software

[Pavel Machek]

On Tue 2009-01-27 12:16:52, Eric W. Biederman wrote:
> "Chris Friesen" <cfriesen@nortel.com> writes:
> 
> > Arjan van de Ven wrote:
> >> On Mon, 26 Jan 2009 09:38:13 -0600
> >> "Chris Friesen" <cfriesen@nortel.com> wrote:
> >>
> >>> Someone is asking me about the feasability of "scrubbing" system
> >>> memory by accessing each page and handling the ECC faults.
> >>>
> >>
> >> Hi,
> >>
> >> I would suggest that you look at the "edac" subsystem, which tries to
> >> do exactly this....
> 
> 
> > edac appears to currently be able to scrub the specific page where the fault
> > occurred.  This is a useful building block, but doesn't provide the ability to
> > march through all of physical memory.
> 
> Well that is the tricky part.  The rest is simply finding which physical
> addresses are valid.  Either by querying the memory controller or looking
> at the range the BIOS gave us.
> 
> That part should not be too hard.  I think it simply has not been implemented
> yet as most ECC chipsets implement this in hardware today.

You can do the scrubbing today by echo reboot > /sys/power/disk; echo
disk > /sys/power/state :-)... or using uswsusp APIs.

Take a look at hibernation code for 'walk all memory' examples...  

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: marching through all physical memory in software

[Nigel Cunningham]

Hi.

On Wed, 2009-01-28 at 20:38 +0100, Pavel Machek wrote:
> You can do the scrubbing today by echo reboot > /sys/power/disk; echo
> disk > /sys/power/state :-)... or using uswsusp APIs.

That won't work. The RAM retains it's contents across a reboot, and even
for a little while after powering off.

Regards,

Nigel

Re: marching through all physical memory in software

[Pavel Machek]

> Hi.
> 
> On Wed, 2009-01-28 at 20:38 +0100, Pavel Machek wrote:
> > You can do the scrubbing today by echo reboot > /sys/power/disk; echo
> > disk > /sys/power/state :-)... or using uswsusp APIs.
> 
> That won't work. The RAM retains it's contents across a reboot, and even
> for a little while after powering off.

Yes, and the original goal was to rewrite all the memory with same
contents so that parity errors don't accumulate. SO scrubbing here !=
trying to clear it.


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: marching through all physical memory in software

[Nigel Cunningham]

Hi again.

On Fri, 2009-01-30 at 10:13 +0100, Pavel Machek wrote:
> > Hi.
> > 
> > On Wed, 2009-01-28 at 20:38 +0100, Pavel Machek wrote:
> > > You can do the scrubbing today by echo reboot > /sys/power/disk; echo
> > > disk > /sys/power/state :-)... or using uswsusp APIs.
> > 
> > That won't work. The RAM retains its contents across a reboot, and even
> > for a little while after powering off.
> 
> Yes, and the original goal was to rewrite all the memory with same
> contents so that parity errors don't accumulate. SO scrubbing here !=
> trying to clear it.

Sorry - I think I missed something.

AFAICS, hibernating is going to be a noop as far as doing anything to
memory that's not touched by the process of hibernating goes. It won't
clear it or scrub it or anything else.

Regards,

Nigel

Re: marching through all physical memory in software

[Andi Kleen]

"Chris Friesen" <cfriesen@nortel.com> writes:

> Someone is asking me about the feasability of "scrubbing" system
> memory by accessing each page and handling the ECC faults.

I would expect any ECC RAM capable memory controller designed 
in the last 10-15 years to be able to do this on its own without
software intervention. Is that not true? It is certainly
on x86 (although you sometimes have to change BIOS options)

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: marching through all physical memory in software

[Henrique de Moraes Holschuh]

On Fri, 06 Feb 2009, Andi Kleen wrote:
> "Chris Friesen" <cfriesen@nortel.com> writes:
> > Someone is asking me about the feasability of "scrubbing" system
> > memory by accessing each page and handling the ECC faults.
> 
> I would expect any ECC RAM capable memory controller designed 
> in the last 10-15 years to be able to do this on its own without
> software intervention. Is that not true? It is certainly
> on x86 (although you sometimes have to change BIOS options)

Is that true for the 82875, and other common desktop chipsets that DO
support ECC ram?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh