[patch resend] oom: thaw threads if oom killed thread is frozen before deferring

[patch resend] oom: thaw threads if oom killed thread is frozen before deferring

[David Rientjes]

If a thread has been oom killed and is frozen, thaw it before returning
to the page allocator.  Otherwise, it can stay frozen indefinitely and
no memory will be freed.

Reported-by: Michal Hocko <mhocko@suse.cz>
Signed-off-by: David Rientjes <rientjes@google.com>
---
 mm/oom_kill.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/mm/oom_kill.c b/mm/oom_kill.c
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -318,8 +318,11 @@ static struct task_struct *select_bad_process(unsigned int *ppoints,
 		 * blocked waiting for another task which itself is waiting
 		 * for memory. Is there a better alternative?
 		 */
-		if (test_tsk_thread_flag(p, TIF_MEMDIE))
+		if (test_tsk_thread_flag(p, TIF_MEMDIE)) {
+			if (unlikely(frozen(p)))
+				thaw_process(p);
 			return ERR_PTR(-1UL);
+		}
 		if (!p->mm)
 			continue;
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[David Rientjes]

If a thread has been oom killed and is frozen, thaw it before returning
to the page allocator.  Otherwise, it can stay frozen indefinitely and
no memory will be freed.

Reported-by: Michal Hocko <mhocko@suse.cz>
Signed-off-by: David Rientjes <rientjes@google.com>
---
 v2: adds the missing header file include, the resend patch was based on a 
     previous patch from Michal that is no longer needed if this is 
     applied.

 mm/oom_kill.c |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 626303b..d897262 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -32,6 +32,7 @@
 #include <linux/mempolicy.h>
 #include <linux/security.h>
 #include <linux/ptrace.h>
+#include <linux/freezer.h>
 
 int sysctl_panic_on_oom;
 int sysctl_oom_kill_allocating_task;
@@ -317,8 +318,11 @@ static struct task_struct *select_bad_process(unsigned int *ppoints,
 		 * blocked waiting for another task which itself is waiting
 		 * for memory. Is there a better alternative?
 		 */
-		if (test_tsk_thread_flag(p, TIF_MEMDIE))
+		if (test_tsk_thread_flag(p, TIF_MEMDIE)) {
+			if (unlikely(frozen(p)))
+				thaw_process(p);
 			return ERR_PTR(-1UL);
+		}
 		if (!p->mm)
 			continue;
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[KOSAKI Motohiro]

2011/10/7 David Rientjes <rientjes@google.com>:
> If a thread has been oom killed and is frozen, thaw it before returning
> to the page allocator.  Otherwise, it can stay frozen indefinitely and
> no memory will be freed.
>
> Reported-by: Michal Hocko <mhocko@suse.cz>
> Signed-off-by: David Rientjes <rientjes@google.com>
> ---
>  v2: adds the missing header file include, the resend patch was based on a
>     previous patch from Michal that is no longer needed if this is
>     applied.

Looks ok to me.
Michal, do you agree this patch?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[Michal Hocko]

Hi,
sorry for the late reply but I was on vacation last week.

On Sat 08-10-11 03:23:15, KOSAKI Motohiro wrote:
> 2011/10/7 David Rientjes <rientjes@google.com>:
> > If a thread has been oom killed and is frozen, thaw it before returning
> > to the page allocator.  Otherwise, it can stay frozen indefinitely and
> > no memory will be freed.
> >
> > Reported-by: Michal Hocko <mhocko@suse.cz>
> > Signed-off-by: David Rientjes <rientjes@google.com>
> > ---
> >  v2: adds the missing header file include, the resend patch was based on a
> >     previous patch from Michal that is no longer needed if this is
> >     applied.
> 
> Looks ok to me.
> Michal, do you agree this patch?

The patch looks good but we still need other 2 patches
(http://comments.gmane.org/gmane.linux.kernel.mm/68578), right?

Anyway, I thought that we agreed on the other approach suggested by
Tejun (make frozen tasks oom killable without thawing). Even in that
case we want the first patch
(http://permalink.gmane.org/gmane.linux.kernel.mm/68576).
-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[KOSAKI Motohiro]

>> Looks ok to me.
>> Michal, do you agree this patch?
> 
> The patch looks good but we still need other 2 patches
> (http://comments.gmane.org/gmane.linux.kernel.mm/68578), right?
> 
> Anyway, I thought that we agreed on the other approach suggested by
> Tejun (make frozen tasks oom killable without thawing). Even in that
> case we want the first patch
> (http://permalink.gmane.org/gmane.linux.kernel.mm/68576).

I'm sorry. I still don't catch up the above long thread. Could you
please resend your final patch again? I'll review it.

Thank you.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[Michal Hocko]

On Tue 11-10-11 10:50:34, KOSAKI Motohiro wrote:
> >> Looks ok to me.
> >> Michal, do you agree this patch?
> > 
> > The patch looks good but we still need other 2 patches
> > (http://comments.gmane.org/gmane.linux.kernel.mm/68578), right?
> > 
> > Anyway, I thought that we agreed on the other approach suggested by
> > Tejun (make frozen tasks oom killable without thawing). Even in that
> > case we want the first patch
> > (http://permalink.gmane.org/gmane.linux.kernel.mm/68576).
> 
> I'm sorry. I still don't catch up the above long thread. 

To sum up. There are 3 patches flying around at the moment.
http://permalink.gmane.org/gmane.linux.kernel.mm/68576
http://permalink.gmane.org/gmane.linux.kernel.mm/68577
http://permalink.gmane.org/gmane.linux.kernel.mm/68583

They are approaching the problem by thawing oom selected frozen task.

Tejun mentioned his work (sorry I do not have a link to patches) that
should enable direct killing frozen tasks. This would mean that we do
not need any special handling from the OOM code paths AFAIU. This would
be much better of course and I guess we can wait for them for 3.2.

Does this make sense?

> Could you please resend your final patch again? I'll review it.
> 
> Thank you.

-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[KOSAKI Motohiro]

> To sum up. There are 3 patches flying around at the moment.
> http://permalink.gmane.org/gmane.linux.kernel.mm/68576
> http://permalink.gmane.org/gmane.linux.kernel.mm/68577
> http://permalink.gmane.org/gmane.linux.kernel.mm/68583
> 
> They are approaching the problem by thawing oom selected frozen task.
> 
> Tejun mentioned his work (sorry I do not have a link to patches) that
> should enable direct killing frozen tasks. This would mean that we do
> not need any special handling from the OOM code paths AFAIU. This would
> be much better of course and I guess we can wait for them for 3.2.
> 
> Does this make sense?

I don't find any bad in the idea. So, I have two questions.

 o Who are writing such patch now?

 o Should we drop current drientjes patch?


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[Michal Hocko]

On Tue 11-10-11 11:57:35, KOSAKI Motohiro wrote:
> > To sum up. There are 3 patches flying around at the moment.
> > http://permalink.gmane.org/gmane.linux.kernel.mm/68576
> > http://permalink.gmane.org/gmane.linux.kernel.mm/68577
> > http://permalink.gmane.org/gmane.linux.kernel.mm/68583
> > 
> > They are approaching the problem by thawing oom selected frozen task.
> > 
> > Tejun mentioned his work (sorry I do not have a link to patches) that
> > should enable direct killing frozen tasks. This would mean that we do
> > not need any special handling from the OOM code paths AFAIU. This would
> > be much better of course and I guess we can wait for them for 3.2.
> > 
> > Does this make sense?
> 
> I don't find any bad in the idea. So, I have two questions.
> 
>  o Who are writing such patch now?

Tejun, could you send a link to those patches, please?

>  o Should we drop current drientjes patch?

This pretty much depends on when Tejun is able to provide the mentioned
patches. If we want to keep David's patch we want also the other
patches, don't we?

-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[David Rientjes]

On Tue, 11 Oct 2011, Michal Hocko wrote:

> The patch looks good but we still need other 2 patches
> (http://comments.gmane.org/gmane.linux.kernel.mm/68578), right?
> 

For the lguest patch, Rusty is the maintainer and has already acked the 
patch, so I think it should be merged through him.  I don't see a need for 
the second patch since we'll now detect frozen oom killed tasks on retry 
and don't need to kill them directly when oom killed (it just adds 
additional, unnecessary code).

> Anyway, I thought that we agreed on the other approach suggested by
> Tejun (make frozen tasks oom killable without thawing). Even in that
> case we want the first patch
> (http://permalink.gmane.org/gmane.linux.kernel.mm/68576).

If that's possible, then we can just add Tejun to add a follow-up patch to 
remove the thaw directly in the oom killer.  I'm thinking that won't be 
possible for 3.2, though, so I don't know why we'd remove 
oom-thaw-threads-if-oom-killed-thread-is-frozen-before-deferring.patch 
from -mm?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[Tejun Heo]

Hello,

On Tue, Oct 11, 2011 at 04:36:28PM -0700, David Rientjes wrote:
> On Tue, 11 Oct 2011, Michal Hocko wrote:
> 
> > The patch looks good but we still need other 2 patches
> > (http://comments.gmane.org/gmane.linux.kernel.mm/68578), right?
> > 
> 
> For the lguest patch, Rusty is the maintainer and has already acked the 
> patch, so I think it should be merged through him.  I don't see a need for 
> the second patch since we'll now detect frozen oom killed tasks on retry 
> and don't need to kill them directly when oom killed (it just adds 
> additional, unnecessary code).
> 
> > Anyway, I thought that we agreed on the other approach suggested by
> > Tejun (make frozen tasks oom killable without thawing). Even in that
> > case we want the first patch
> > (http://permalink.gmane.org/gmane.linux.kernel.mm/68576).
> 
> If that's possible, then we can just add Tejun to add a follow-up patch to 
> remove the thaw directly in the oom killer.  I'm thinking that won't be 
> possible for 3.2, though, so I don't know why we'd remove 
> oom-thaw-threads-if-oom-killed-thread-is-frozen-before-deferring.patch 
> from -mm?

Yeah, it's a bit unclear.  All (or at least most) patches which were
necessary for this patch was queued in Rafael's tree before korg went
belly up and then we both lost track of the tree and I need to
regenerate the tree and ask Rafael to pull again.  I think the merge
window is already too close, so please go ahead with the acked fix.
Let's clean it up during the next devel cycle.

Thank you.

-- 
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[David Rientjes]

On Tue, 11 Oct 2011, Tejun Heo wrote:

> > If that's possible, then we can just add Tejun to add a follow-up patch to 
> > remove the thaw directly in the oom killer.  I'm thinking that won't be 
> > possible for 3.2, though, so I don't know why we'd remove 
> > oom-thaw-threads-if-oom-killed-thread-is-frozen-before-deferring.patch 
> > from -mm?
> 
> Yeah, it's a bit unclear.  All (or at least most) patches which were
> necessary for this patch was queued in Rafael's tree before korg went
> belly up and then we both lost track of the tree and I need to
> regenerate the tree and ask Rafael to pull again.

Eek, what a pain.

> I think the merge
> window is already too close, so please go ahead with the acked fix.
> Let's clean it up during the next devel cycle.
> 

Ok, sounds good.  When frozen oom killed threads can always move down the 
exit path, then we can just remember to remove the thaw_process() from the 
oom killer.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[Michal Hocko]

On Tue 11-10-11 16:36:28, David Rientjes wrote:
> On Tue, 11 Oct 2011, Michal Hocko wrote:
> 
> > The patch looks good but we still need other 2 patches
> > (http://comments.gmane.org/gmane.linux.kernel.mm/68578), right?
> > 
> 
> For the lguest patch, Rusty is the maintainer and has already acked the 
> patch, so I think it should be merged through him.  I don't see a need for 
> the second patch since we'll now detect frozen oom killed tasks on retry 
> and don't need to kill them directly when oom killed (it just adds 
> additional, unnecessary code).

OK, my understanding was that we need both patches, but you are right,
the later one should be sufficient.

> 
> > Anyway, I thought that we agreed on the other approach suggested by
> > Tejun (make frozen tasks oom killable without thawing). Even in that
> > case we want the first patch
> > (http://permalink.gmane.org/gmane.linux.kernel.mm/68576).
> 
> If that's possible, then we can just add Tejun to add a follow-up patch to 
> remove the thaw directly in the oom killer.  

OK

> I'm thinking that won't be possible for 3.2, though, so I don't know why we'd
> remove oom-thaw-threads-if-oom-killed-thread-is-frozen-before-deferring.patch
> from -mm?

No need for that then.

Thanks
-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch v2] oom: thaw threads if oom killed thread is frozen before deferring

[Michal Hocko]

On Wed 12-10-11 08:50:26, Michal Hocko wrote:
> On Tue 11-10-11 16:36:28, David Rientjes wrote:
> > On Tue, 11 Oct 2011, Michal Hocko wrote:
> > 
> > > The patch looks good but we still need other 2 patches
> > > (http://comments.gmane.org/gmane.linux.kernel.mm/68578), right?
> > > 
> > 
> > For the lguest patch, Rusty is the maintainer and has already acked the 
> > patch, so I think it should be merged through him.  I don't see a need for 
> > the second patch since we'll now detect frozen oom killed tasks on retry 
> > and don't need to kill them directly when oom killed (it just adds 
> > additional, unnecessary code).
> 
> OK, my understanding was that we need both patches, but you are right,
> the later one should be sufficient.

Ahh and I forgot to add:
Acked-by: Michal Hocko <mhocko@suse.cz>

If it matters.

-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch resend] oom: thaw threads if oom killed thread is frozen before deferring

[Oleg Nesterov]

I guess this patch doesn't need my ack, but just in case, I think it is
fine. Even if (perhaps) we can do something better later, with the upcoming
changes in refrigerator.

David. Could you also resend you patches which remove the (imho really
annoying) mm->oom_disable_count? Feel free to add my ack or reviewed-by.

Oleg.

On 10/07, David Rientjes wrote:
>
> If a thread has been oom killed and is frozen, thaw it before returning
> to the page allocator.  Otherwise, it can stay frozen indefinitely and
> no memory will be freed.
>
> Reported-by: Michal Hocko <mhocko@suse.cz>
> Signed-off-by: David Rientjes <rientjes@google.com>
> ---
>  mm/oom_kill.c |    5 ++++-
>  1 files changed, 4 insertions(+), 1 deletions(-)
>
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -318,8 +318,11 @@ static struct task_struct *select_bad_process(unsigned int *ppoints,
>  		 * blocked waiting for another task which itself is waiting
>  		 * for memory. Is there a better alternative?
>  		 */
> -		if (test_tsk_thread_flag(p, TIF_MEMDIE))
> +		if (test_tsk_thread_flag(p, TIF_MEMDIE)) {
> +			if (unlikely(frozen(p)))
> +				thaw_process(p);
>  			return ERR_PTR(-1UL);
> +		}
>  		if (!p->mm)
>  			continue;
>  

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch resend] oom: thaw threads if oom killed thread is frozen before deferring

[David Rientjes]

On Tue, 11 Oct 2011, Oleg Nesterov wrote:

> David. Could you also resend you patches which remove the (imho really
> annoying) mm->oom_disable_count? Feel free to add my ack or reviewed-by.
> 

As far as I know (I can't confirm because userweb.kernel.org is still 
down), oom-remove-oom_disable_count.patch is still in the -mm tree.  It 
was merged September 2 so I believe it's 3.2 material.

Andrew, please add Oleg's reviewed-by to that patch (in addition to the 
reported-by which already exists) if it's still merged.  Otherwise, please 
let me know and I'll resend it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch resend] oom: thaw threads if oom killed thread is frozen before deferring

[Andrew Morton]

On Tue, 11 Oct 2011 16:30:26 -0700 (PDT)
David Rientjes <rientjes@google.com> wrote:

> On Tue, 11 Oct 2011, Oleg Nesterov wrote:
> 
> > David. Could you also resend you patches which remove the (imho really
> > annoying) mm->oom_disable_count? Feel free to add my ack or reviewed-by.
> > 
> 
> As far as I know (I can't confirm because userweb.kernel.org is still 
> down), oom-remove-oom_disable_count.patch is still in the -mm tree.  It 
> was merged September 2 so I believe it's 3.2 material.

yup.  It's in linux-next too.

> Andrew, please add Oleg's reviewed-by to that patch (in addition to the 
> reported-by which already exists) if it's still merged.

Done, thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch resend] oom: thaw threads if oom killed thread is frozen before deferring

[Oleg Nesterov]

On 10/11, David Rientjes wrote:
>
> On Tue, 11 Oct 2011, Oleg Nesterov wrote:
>
> > David. Could you also resend you patches which remove the (imho really
> > annoying) mm->oom_disable_count? Feel free to add my ack or reviewed-by.
> >
>
> As far as I know (I can't confirm because userweb.kernel.org is still
> down), oom-remove-oom_disable_count.patch is still in the -mm tree.  It
> was merged September 2 so I believe it's 3.2 material.

Ah, great. Somehow I thought it was missed.

Thanks David,

Oleg.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>