aio-stress throughput regressions from 2.6.11 to 2.6.12

aio-stress throughput regressions from 2.6.11 to 2.6.12

[Suparna Bhattacharya]

Has anyone else noticed major throughput regressions for random
reads/writes with aio-stress in 2.6.12 ?
Or have there been any other FS/IO regressions lately ?

On one test system I see a degradation from around 17+ MB/s to 11MB/s
for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
to 2.6.12. It doesn't seem filesystem specific. Not good :(

BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact
(I copied those back to my 2.6.11 tree and tried the runs with no effect)
So it is something else ...

Ideas/thoughts/observations ?

Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

aio-stress throughput regressions from 2.6.11 to 2.6.12

[Suparna Bhattacharya]

Has anyone else noticed major throughput regressions for random
reads/writes with aio-stress in 2.6.12 ?
Or have there been any other FS/IO regressions lately ?

On one test system I see a degradation from around 17+ MB/s to 11MB/s
for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
to 2.6.12. It doesn't seem filesystem specific. Not good :(

BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact
(I copied those back to my 2.6.11 tree and tried the runs with no effect)
So it is something else ...

Ideas/thoughts/observations ?

Regards
Suparna

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Benjamin LaHaise]

On Fri, Jul 01, 2005 at 01:26:00PM +0530, Suparna Bhattacharya wrote:
> On one test system I see a degradation from around 17+ MB/s to 11MB/s
> for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> to 2.6.12. It doesn't seem filesystem specific. Not good :(

What sort of io subsystem does it have?  Also, does changing the 
elevator make any difference?  I'm away for the long weekend, but will 
help look into this next week.

		-ben

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Suparna Bhattacharya]

On Fri, Jul 01, 2005 at 10:25:55AM -0400, Benjamin LaHaise wrote:
> On Fri, Jul 01, 2005 at 01:26:00PM +0530, Suparna Bhattacharya wrote:
> > On one test system I see a degradation from around 17+ MB/s to 11MB/s
> > for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> > to 2.6.12. It doesn't seem filesystem specific. Not good :(
> 
> What sort of io subsystem does it have?  Also, does changing the 

aic7xxx.
> elevator make any difference?  I'm away for the long weekend, but will 

I tried switching to the noop elevator - the regression was still there.

> help look into this next week.

Do you see the regression as well, or is it just me ?

Regards
Suparna

> 
> 		-ben

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Benjamin LaHaise]

On Wed, Jul 06, 2005 at 10:00:56AM +0530, Suparna Bhattacharya wrote:
> Do you see the regression as well, or is it just me ?

Throughput for me on an ICH6 SATA system using O_DIRECT seems to remain 
pretty constant across 2.6.11-something-FC3 to 2.6.13-rc2 around 31MB/s.  
2.6.11 proper hangs for me, though.

		-ben
-- 
"Time is what keeps everything from happening all at once." -- John Wheeler

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Benjamin LaHaise]

On Wed, Jul 06, 2005 at 10:00:56AM +0530, Suparna Bhattacharya wrote:
> Do you see the regression as well, or is it just me ?

Throughput for me on an ICH6 SATA system using O_DIRECT seems to remain 
pretty constant across 2.6.11-something-FC3 to 2.6.13-rc2 around 31MB/s.  
2.6.11 proper hangs for me, though.

		-ben
-- 
"Time is what keeps everything from happening all at once." -- John Wheeler
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Chris Mason]

On Friday 01 July 2005 03:56, Suparna Bhattacharya wrote:
> Has anyone else noticed major throughput regressions for random
> reads/writes with aio-stress in 2.6.12 ?
> Or have there been any other FS/IO regressions lately ?
>
> On one test system I see a degradation from around 17+ MB/s to 11MB/s
> for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> to 2.6.12. It doesn't seem filesystem specific. Not good :(
>
> BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an
> impact (I copied those back to my 2.6.11 tree and tried the runs with no
> effect) So it is something else ...
>
> Ideas/thoughts/observations ?

Lets try to narrow it down a bit:

aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request 
at a time).  This doesn't take the aio subsystem out of the picture, but it 
does make the block layer interaction more or less the same as non-aio 
benchmarks.  If this is slow, I would suspect something in the block layer, 
and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.

If it doesn't regress, I would suspect something in the aio core.  My first 
attempts at the context switch reduction patches caused this kind of 
regression.  There was too much latency in sending the events up to userland.

Other options:

Try different elevators
Try O_SYNC aio random writes
Try aio random reads
Try buffers random reads

-chris

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Chris Mason]

On Friday 01 July 2005 03:56, Suparna Bhattacharya wrote:
> Has anyone else noticed major throughput regressions for random
> reads/writes with aio-stress in 2.6.12 ?
> Or have there been any other FS/IO regressions lately ?
>
> On one test system I see a degradation from around 17+ MB/s to 11MB/s
> for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> to 2.6.12. It doesn't seem filesystem specific. Not good :(
>
> BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an
> impact (I copied those back to my 2.6.11 tree and tried the runs with no
> effect) So it is something else ...
>
> Ideas/thoughts/observations ?

Lets try to narrow it down a bit:

aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request 
at a time).  This doesn't take the aio subsystem out of the picture, but it 
does make the block layer interaction more or less the same as non-aio 
benchmarks.  If this is slow, I would suspect something in the block layer, 
and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.

If it doesn't regress, I would suspect something in the aio core.  My first 
attempts at the context switch reduction patches caused this kind of 
regression.  There was too much latency in sending the events up to userland.

Other options:

Try different elevators
Try O_SYNC aio random writes
Try aio random reads
Try buffers random reads

-chris
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Benjamin LaHaise]

On Tue, Jul 05, 2005 at 10:00:24AM -0400, Chris Mason wrote:
> If it doesn't regress, I would suspect something in the aio core.  My first 
> attempts at the context switch reduction patches caused this kind of 
> regression.  There was too much latency in sending the events up to userland.

AIO will by its very nature have a higher rate of context switches unless 
the subsystem in question does its completion from the interrupt handler.  
There are a few optimizations in this area that we can do to improve things 
for some of the common usage models: when sleeping in read_events() we can 
do iocb runs in the sleeping task instead of switching to the aio helper 
daemon.  That might help for Oracle's usage model of aio, and it should 
also help aio-stress.

There are also other ways of reducing the overhead of the context switches.  
O_DIRECT operations could take note of the mm they are operating on and 
do its get_user_pages() on the mm without the tlb being flushed.

		-ben
-- 
"Time is what keeps everything from happening all at once." -- John Wheeler

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Benjamin LaHaise]

On Tue, Jul 05, 2005 at 10:00:24AM -0400, Chris Mason wrote:
> If it doesn't regress, I would suspect something in the aio core.  My first 
> attempts at the context switch reduction patches caused this kind of 
> regression.  There was too much latency in sending the events up to userland.

AIO will by its very nature have a higher rate of context switches unless 
the subsystem in question does its completion from the interrupt handler.  
There are a few optimizations in this area that we can do to improve things 
for some of the common usage models: when sleeping in read_events() we can 
do iocb runs in the sleeping task instead of switching to the aio helper 
daemon.  That might help for Oracle's usage model of aio, and it should 
also help aio-stress.

There are also other ways of reducing the overhead of the context switches.  
O_DIRECT operations could take note of the mm they are operating on and 
do its get_user_pages() on the mm without the tlb being flushed.

		-ben
-- 
"Time is what keeps everything from happening all at once." -- John Wheeler
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Suparna Bhattacharya]

On Tue, Jul 05, 2005 at 10:00:24AM -0400, Chris Mason wrote:
> On Friday 01 July 2005 03:56, Suparna Bhattacharya wrote:
> > Has anyone else noticed major throughput regressions for random
> > reads/writes with aio-stress in 2.6.12 ?
> > Or have there been any other FS/IO regressions lately ?
> >
> > On one test system I see a degradation from around 17+ MB/s to 11MB/s
> > for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> > to 2.6.12. It doesn't seem filesystem specific. Not good :(
> >
> > BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an
> > impact (I copied those back to my 2.6.11 tree and tried the runs with no
> > effect) So it is something else ...
> >
> > Ideas/thoughts/observations ?
> 
> Lets try to narrow it down a bit:
> 
> aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request 
This doesn't regress - the problem really happens when we don't wait one
at a time.

> at a time).  This doesn't take the aio subsystem out of the picture, but it 
> does make the block layer interaction more or less the same as non-aio 
> benchmarks.  If this is slow, I would suspect something in the block layer, 
> and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.
> 
> If it doesn't regress, I would suspect something in the aio core.  My first 
> attempts at the context switch reduction patches caused this kind of 
> regression.  There was too much latency in sending the events up to userland.
> 
> Other options:
> 
> Try different elevators

Still regresses (I tried noop instead of as)

> Try O_SYNC aio random writes
> Try aio random reads
> Try buffers random reads

Again all these end up waiting one at a time with mainline because it
forces buffered AIO to be synchronous, so we the regression doesn't
show up. But, when I apply my patches to make buffered fsAIO async,
so we aren't waiting one at a time -- there is a similar regression.
In fact it was this regression that made me go back and check if it
was happening with AIO-DIO as well.

Regards
Suparna

> 
> -chris

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Suparna Bhattacharya]

On Tue, Jul 05, 2005 at 10:00:24AM -0400, Chris Mason wrote:
> On Friday 01 July 2005 03:56, Suparna Bhattacharya wrote:
> > Has anyone else noticed major throughput regressions for random
> > reads/writes with aio-stress in 2.6.12 ?
> > Or have there been any other FS/IO regressions lately ?
> >
> > On one test system I see a degradation from around 17+ MB/s to 11MB/s
> > for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> > to 2.6.12. It doesn't seem filesystem specific. Not good :(
> >
> > BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an
> > impact (I copied those back to my 2.6.11 tree and tried the runs with no
> > effect) So it is something else ...
> >
> > Ideas/thoughts/observations ?
> 
> Lets try to narrow it down a bit:
> 
> aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request 
This doesn't regress - the problem really happens when we don't wait one
at a time.

> at a time).  This doesn't take the aio subsystem out of the picture, but it 
> does make the block layer interaction more or less the same as non-aio 
> benchmarks.  If this is slow, I would suspect something in the block layer, 
> and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.
> 
> If it doesn't regress, I would suspect something in the aio core.  My first 
> attempts at the context switch reduction patches caused this kind of 
> regression.  There was too much latency in sending the events up to userland.
> 
> Other options:
> 
> Try different elevators

Still regresses (I tried noop instead of as)

> Try O_SYNC aio random writes
> Try aio random reads
> Try buffers random reads

Again all these end up waiting one at a time with mainline because it
forces buffered AIO to be synchronous, so we the regression doesn't
show up. But, when I apply my patches to make buffered fsAIO async,
so we aren't waiting one at a time -- there is a similar regression.
In fact it was this regression that made me go back and check if it
was happening with AIO-DIO as well.

Regards
Suparna

> 
> -chris

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

Re: aio-stress regressions in 2.6.12 narrowed down to AIC7xxx

[Suparna Bhattacharya]

OK, I think I have narrowed it down to aic7xxx driver, because when
I specify CONFIG_SCSI_AIC7XXX_OLD, it does not regress. 

The regression occurs from 2.6.11 to 2.6.12  (from 17MB/s it goes down to
11MB/s)

The regression is still there in 2.6.13-rc1 +  the "speed fix" patch(discussed
in the recent aic7xxx regression thread on linux-scsi)

Recreate by running: aio-stress -O -o3 <1 GB testfile>

Config options:
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y


Comparing dmesg outputs
On 2.6.11
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
        <Adaptec aic7896/97 Ultra2 SCSI adapter>
        aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs

(scsi0:A:0): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
  Vendor: IBM-ESXS  Model: ST318305LC    !#  Rev: B245
  Type:   Direct-Access                      ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
(scsi0:A:1): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
  Vendor: IBM-ESXS  Model: ST318305LC    !#  Rev: B245
...

On 2.6.12
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
        <Adaptec aic7896/97 Ultra2 SCSI adapter>
        aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs

  Vendor: IBM-ESXS  Model: ST318305LC    !#  Rev: B245
  Type:   Direct-Access                      ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
 target0:0:0: Beginning Domain Validation
WIDTH IS 1
(scsi0:A:0): 6.600MB/s transfers (16bit)
(scsi0:A:0): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
 target0:0:0: Ending Domain Validation
  Vendor: IBM-ESXS  Model: ST318305LC    !#  Rev: B245
...


On 2.6.13-rc1 + "speed fix"
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
        <Adaptec aic7896/97 Ultra2 SCSI adapter>
        aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs

 target0:0:0: asynchronous.
  Vendor: IBM-ESXS  Model: ST318305LC    !#  Rev: B245
  Type:   Direct-Access                      ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
 target0:0:0: Beginning Domain Validation
 target0:0:0: wide asynchronous.
 target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 63)
 target0:0:0: Ending Domain Validation
target0:0:1: asynchronous.
  Vendor: IBM-ESXS  Model: ST318305LC    !#  Rev: B245
...

Regards
Suparna


On Wed, Jul 06, 2005 at 04:07:29PM +0530, Suparna Bhattacharya wrote:
> On Tue, Jul 05, 2005 at 10:00:24AM -0400, Chris Mason wrote:
> > On Friday 01 July 2005 03:56, Suparna Bhattacharya wrote:
> > > Has anyone else noticed major throughput regressions for random
> > > reads/writes with aio-stress in 2.6.12 ?
> > > Or have there been any other FS/IO regressions lately ?
> > >
> > > On one test system I see a degradation from around 17+ MB/s to 11MB/s
> > > for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> > > to 2.6.12. It doesn't seem filesystem specific. Not good :(
> > >
> > > BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an
> > > impact (I copied those back to my 2.6.11 tree and tried the runs with no
> > > effect) So it is something else ...
> > >
> > > Ideas/thoughts/observations ?
> > 
> > Lets try to narrow it down a bit:
> > 
> > aio-stress -o 3 -d 1 will set the depth to 1, (io_submit then wait one request 
> This doesn't regress - the problem really happens when we don't wait one
> at a time.
> 
> > at a time).  This doesn't take the aio subsystem out of the picture, but it 
> > does make the block layer interaction more or less the same as non-aio 
> > benchmarks.  If this is slow, I would suspect something in the block layer, 
> > and iozone -I -i 2 -w -f testext3/rwfile5 should also show the regression.
> > 
> > If it doesn't regress, I would suspect something in the aio core.  My first 
> > attempts at the context switch reduction patches caused this kind of 
> > regression.  There was too much latency in sending the events up to userland.
> > 
> > Other options:
> > 
> > Try different elevators
> 
> Still regresses (I tried noop instead of as)
> 
> > Try O_SYNC aio random writes
> > Try aio random reads
> > Try buffers random reads
> 
> Again all these end up waiting one at a time with mainline because it
> forces buffered AIO to be synchronous, so we the regression doesn't
> show up. But, when I apply my patches to make buffered fsAIO async,
> so we aren't waiting one at a time -- there is a similar regression.
> In fact it was this regression that made me go back and check if it
> was happening with AIO-DIO as well.
> 
> Regards
> Suparna
> 
> > 
> > -chris
> 
> -- 
> Suparna Bhattacharya (suparna@in.ibm.com)
> Linux Technology Center
> IBM Software Lab, India
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-aio' in
> the body to majordomo@kvack.org.  For more info on Linux AIO,
> see: http://www.kvack.org/aio/
> Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Lab, India

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Sébastien Dugué]

On Fri, 2005-07-01 at 13:26 +0530, Suparna Bhattacharya wrote:
> Has anyone else noticed major throughput regressions for random
> reads/writes with aio-stress in 2.6.12 ?
> Or have there been any other FS/IO regressions lately ?
> 
> On one test system I see a degradation from around 17+ MB/s to 11MB/s
> for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> to 2.6.12. It doesn't seem filesystem specific. Not good :(
> 
> BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact
> (I copied those back to my 2.6.11 tree and tried the runs with no effect)
> So it is something else ...
> 
> Ideas/thoughts/observations ?
> 
> Regards
> Suparna
> 

  I'm too seeing a regression, but between 2.6.10 and 2.6.12 and using
sysbench + MySQL (with POSIX AIO). The difference is roughly 8% slower
for 2.6.12. I'm currently trying to trace it.

  aio-stress shows no difference so it probably does not come from
kernel AIO.

  Sébastien.


-- 
------------------------------------------------------

  Sébastien Dugué                BULL/FREC:B1-247
  phone: (+33) 476 29 77 70      Bullcom: 229-7770

  mailto:sebastien.dugue@bull.net

  Linux POSIX AIO: http://www.bullopensource.org/posix
  
------------------------------------------------------

Re: aio-stress throughput regressions from 2.6.11 to 2.6.12

[Sébastien Dugué]

On Fri, 2005-07-01 at 13:26 +0530, Suparna Bhattacharya wrote:
> Has anyone else noticed major throughput regressions for random
> reads/writes with aio-stress in 2.6.12 ?
> Or have there been any other FS/IO regressions lately ?
> 
> On one test system I see a degradation from around 17+ MB/s to 11MB/s
> for random O_DIRECT AIO (aio-stress -o3 testext3/rwfile5) from 2.6.11
> to 2.6.12. It doesn't seem filesystem specific. Not good :(
> 
> BTW, Chris/Ben, it doesn't look like the changes to aio.c have had an impact
> (I copied those back to my 2.6.11 tree and tried the runs with no effect)
> So it is something else ...
> 
> Ideas/thoughts/observations ?
> 
> Regards
> Suparna
> 

  I'm too seeing a regression, but between 2.6.10 and 2.6.12 and using
sysbench + MySQL (with POSIX AIO). The difference is roughly 8% slower
for 2.6.12. I'm currently trying to trace it.

  aio-stress shows no difference so it probably does not come from
kernel AIO.

  Sébastien.


-- 
------------------------------------------------------

  Sébastien Dugué                BULL/FREC:B1-247
  phone: (+33) 476 29 77 70      Bullcom: 229-7770

  mailto:sebastien.dugue@bull.net

  Linux POSIX AIO: http://www.bullopensource.org/posix
  
------------------------------------------------------

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>