[LTP] [PATCH] realtime: Fix the pass criterion of pi_perf test case.

[LTP] [PATCH] realtime: Fix the pass criterion of pi_perf test case.

[Sripathi Kodi]

Hi,

The pass criterion in pi_perf test case is wrong. It compares the minimum 
amount of time taken by the low priority thread with the maximum amount of 
time taken by the high priority thread to calculate the PI delay. Obviously, 
these min and max don't necessarily happen in the same iteration, resulting in 
a number of false failures. The correct way is to compare the time taken by 
low and high priority threads in each iteration and then find the maximum delay 
experienced by high priority thread across the iterations. This patch 
implements the change. Additionally, this patch removes lock_wait_dat array, 
which is not needed anymore as well as makes a couple of messages easier to 
understand.

This patch changes the messages displayed by this test case slightly. They 
look like the following now:

Low prio thread started
High prio thread started
Busy 0 started
Busy 1 started
Busy 2 started
Busy 3 started
Time taken for high prio thread to get the lock once released by low prio 
thread
Min delay = 10 us
Max delay = 36 us
Average delay = 17.06 us
Standard Deviation = 8.11 us
Quantiles:
99.0% < 36
Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
Result: PASS

I have tested this by running 1000 iterations of pi_perf test on a couple of 
machines.

Thanks,
Sripathi.

---

realtime: Fix the pass criterion of pi_perf test case.

Signed-off-by: Sripathi Kodi <sripathik@in.ibm.com>
Acked-by: Will Schmidt <will_schmidt@vnet.ibm.com>
Acked-by: Gowrishankar <gowrishankar.m@linux.vnet.ibm.com>


diff -uprN ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c
--- ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c	2009-07-03 12:21:41.000000000 +0530
+++ ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c	2009-07-03 12:24:44.000000000 +0530
@@ -1,6 +1,6 @@
 /******************************************************************************
  *
- *   Copyright © International Business Machines  Corp., 2007, 2008
+ *   Copyright © International Business Machines  Corp., 2007, 2008, 2009
  *
  *   This program is free software;  you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -36,6 +36,8 @@
  *
  * HISTORY
  *     2007-Nov-20:    Initial version by Sripathi Kodi <sripathik@in.ibm.com>
+ *     2009-Jul-03:    Pass criteria corrected by Sripathi Kodi
+ *                                                      <sripathik@in.ibm.com>
  *
  *****************************************************************************/
 
@@ -71,9 +73,9 @@ static unsigned int high_work_time = DEF
 static unsigned int busy_work_time;
 static int num_busy = -1;
 
-nsec_t low_unlock, high_get_lock;
+nsec_t low_unlock, max_pi_delay;
 
-stats_container_t lock_wait_dat, low_dat, cpu_delay_dat;
+stats_container_t low_dat, cpu_delay_dat;
 stats_container_t cpu_delay_hist;
 stats_quantiles_t cpu_delay_quantiles;
 
@@ -165,10 +167,9 @@ void * low_prio_thread(void *arg)
 
 void * high_prio_thread(void *arg)
 {
-	nsec_t high_start, high_spent, high_end;
+	nsec_t high_start, high_end, high_get_lock;
 	unsigned int i;
 
-	stats_container_init(&lock_wait_dat, iterations);
 	stats_container_init(&cpu_delay_dat, iterations);
 	stats_container_init(&cpu_delay_hist, HIST_BUCKETS);
 	stats_quantiles_init(&cpu_delay_quantiles, (int)log10(iterations));
@@ -184,14 +185,11 @@ void * high_prio_thread(void *arg)
 		high_start = rt_gettime();
 		pthread_mutex_lock(&lock);
 		high_end = rt_gettime();
-		high_spent = high_end - high_start;
 		high_get_lock = high_end - low_unlock;
 
 		busy_work_ms(high_work_time);
 		pthread_mutex_unlock(&lock);
 
-		lock_wait_dat.records[i].x = i;
-		lock_wait_dat.records[i].y = high_spent / NS_PER_US;
 		cpu_delay_dat.records[i].x = i;
 		cpu_delay_dat.records[i].y = high_get_lock / NS_PER_US;
 
@@ -206,14 +204,16 @@ void * high_prio_thread(void *arg)
 				"Latency (us)", "Samples", &cpu_delay_hist, "steps");
 
 	printf("Time taken for high prio thread to get the lock once released by low prio thread\n");
-	printf("Min wait time = %ld us\n", stats_min(&cpu_delay_dat));
-	printf("Max wait time = %ld us\n", stats_max(&cpu_delay_dat));
-	printf("Average wait time = %4.2f us\n", stats_avg(&cpu_delay_dat));
+	printf("Min delay = %ld us\n", stats_min(&cpu_delay_dat));
+	printf("Max delay = %ld us\n", stats_max(&cpu_delay_dat));
+	printf("Average delay = %4.2f us\n", stats_avg(&cpu_delay_dat));
 	printf("Standard Deviation = %4.2f us\n", stats_stddev(&cpu_delay_dat));
 	printf("Quantiles:\n");
 	stats_quantiles_calc(&cpu_delay_dat, &cpu_delay_quantiles);
 	stats_quantiles_print(&cpu_delay_quantiles);
 
+	max_pi_delay = stats_max(&cpu_delay_dat);
+
 	return NULL;
 }
 
@@ -261,13 +261,11 @@ int main(int argc, char *argv[])
 	}
 
 	join_threads();
-	printf("Low prio lock held time (min) = %ld us\n", stats_min(&low_dat));
-	printf("High prio lock wait time (max) = %ld us\n", stats_max(&lock_wait_dat));
 	printf("Criteria: High prio lock wait time < "
 			"(Low prio lock held time + %d us)\n", (int)pass_criteria);
 
 	ret = 0;
-	if (stats_max(&lock_wait_dat) > stats_min(&low_dat) + (int)pass_criteria)
+	if (max_pi_delay > pass_criteria)
 		ret = 1;
 
 	printf("Result: %s\n", ret ? "FAIL" : "PASS");


------------------------------------------------------------------------------
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] [PATCH] realtime: Fix the pass criterion of pi_perf test case.

[Subrata Modak]

On Fri, 2009-07-03 at 12:35 +0530, Sripathi Kodi wrote: 
> Hi,
> 
> The pass criterion in pi_perf test case is wrong. It compares the minimum 
> amount of time taken by the low priority thread with the maximum amount of 
> time taken by the high priority thread to calculate the PI delay. Obviously, 
> these min and max don't necessarily happen in the same iteration, resulting in 
> a number of false failures. The correct way is to compare the time taken by 
> low and high priority threads in each iteration and then find the maximum delay 
> experienced by high priority thread across the iterations. This patch 
> implements the change. Additionally, this patch removes lock_wait_dat array, 
> which is not needed anymore as well as makes a couple of messages easier to 
> understand.
> 
> This patch changes the messages displayed by this test case slightly. They 
> look like the following now:
> 
> Low prio thread started
> High prio thread started
> Busy 0 started
> Busy 1 started
> Busy 2 started
> Busy 3 started
> Time taken for high prio thread to get the lock once released by low prio 
> thread
> Min delay = 10 us
> Max delay = 36 us
> Average delay = 17.06 us
> Standard Deviation = 8.11 us
> Quantiles:
> 99.0% < 36
> Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
> Result: PASS
> 
> I have tested this by running 1000 iterations of pi_perf test on a couple of 
> machines.
> 
> Thanks,
> Sripathi.
> 
> ---
> 
> realtime: Fix the pass criterion of pi_perf test case.
> 
> Signed-off-by: Sripathi Kodi <sripathik@in.ibm.com>

Hi Sri,

Is this based on the latest snapshot. I get the following failures:

patching file testcases/realtime/func/pi_perf/pi_perf.c
Hunk #1 FAILED at 1.


Can you please resend ?

Regards--
Subrata

> Acked-by: Will Schmidt <will_schmidt@vnet.ibm.com>
> Acked-by: Gowrishankar <gowrishankar.m@linux.vnet.ibm.com>
> 
> 
> diff -uprN ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c
> --- ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c	2009-07-03 12:21:41.000000000 +0530
> +++ ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c	2009-07-03 12:24:44.000000000 +0530
> @@ -1,6 +1,6 @@
>  /******************************************************************************
>   *
> - *   Copyright © International Business Machines  Corp., 2007, 2008
> + *   Copyright © International Business Machines  Corp., 2007, 2008, 2009
>   *
>   *   This program is free software;  you can redistribute it and/or modify
>   *   it under the terms of the GNU General Public License as published by
> @@ -36,6 +36,8 @@
>   *
>   * HISTORY
>   *     2007-Nov-20:    Initial version by Sripathi Kodi <sripathik@in.ibm.com>
> + *     2009-Jul-03:    Pass criteria corrected by Sripathi Kodi
> + *                                                      <sripathik@in.ibm.com>
>   *
>   *****************************************************************************/
>  
> @@ -71,9 +73,9 @@ static unsigned int high_work_time = DEF
>  static unsigned int busy_work_time;
>  static int num_busy = -1;
>  
> -nsec_t low_unlock, high_get_lock;
> +nsec_t low_unlock, max_pi_delay;
>  
> -stats_container_t lock_wait_dat, low_dat, cpu_delay_dat;
> +stats_container_t low_dat, cpu_delay_dat;
>  stats_container_t cpu_delay_hist;
>  stats_quantiles_t cpu_delay_quantiles;
>  
> @@ -165,10 +167,9 @@ void * low_prio_thread(void *arg)
>  
>  void * high_prio_thread(void *arg)
>  {
> -	nsec_t high_start, high_spent, high_end;
> +	nsec_t high_start, high_end, high_get_lock;
>  	unsigned int i;
>  
> -	stats_container_init(&lock_wait_dat, iterations);
>  	stats_container_init(&cpu_delay_dat, iterations);
>  	stats_container_init(&cpu_delay_hist, HIST_BUCKETS);
>  	stats_quantiles_init(&cpu_delay_quantiles, (int)log10(iterations));
> @@ -184,14 +185,11 @@ void * high_prio_thread(void *arg)
>  		high_start = rt_gettime();
>  		pthread_mutex_lock(&lock);
>  		high_end = rt_gettime();
> -		high_spent = high_end - high_start;
>  		high_get_lock = high_end - low_unlock;
>  
>  		busy_work_ms(high_work_time);
>  		pthread_mutex_unlock(&lock);
>  
> -		lock_wait_dat.records[i].x = i;
> -		lock_wait_dat.records[i].y = high_spent / NS_PER_US;
>  		cpu_delay_dat.records[i].x = i;
>  		cpu_delay_dat.records[i].y = high_get_lock / NS_PER_US;
>  
> @@ -206,14 +204,16 @@ void * high_prio_thread(void *arg)
>  				"Latency (us)", "Samples", &cpu_delay_hist, "steps");
>  
>  	printf("Time taken for high prio thread to get the lock once released by low prio thread\n");
> -	printf("Min wait time = %ld us\n", stats_min(&cpu_delay_dat));
> -	printf("Max wait time = %ld us\n", stats_max(&cpu_delay_dat));
> -	printf("Average wait time = %4.2f us\n", stats_avg(&cpu_delay_dat));
> +	printf("Min delay = %ld us\n", stats_min(&cpu_delay_dat));
> +	printf("Max delay = %ld us\n", stats_max(&cpu_delay_dat));
> +	printf("Average delay = %4.2f us\n", stats_avg(&cpu_delay_dat));
>  	printf("Standard Deviation = %4.2f us\n", stats_stddev(&cpu_delay_dat));
>  	printf("Quantiles:\n");
>  	stats_quantiles_calc(&cpu_delay_dat, &cpu_delay_quantiles);
>  	stats_quantiles_print(&cpu_delay_quantiles);
>  
> +	max_pi_delay = stats_max(&cpu_delay_dat);
> +
>  	return NULL;
>  }
>  
> @@ -261,13 +261,11 @@ int main(int argc, char *argv[])
>  	}
>  
>  	join_threads();
> -	printf("Low prio lock held time (min) = %ld us\n", stats_min(&low_dat));
> -	printf("High prio lock wait time (max) = %ld us\n", stats_max(&lock_wait_dat));
>  	printf("Criteria: High prio lock wait time < "
>  			"(Low prio lock held time + %d us)\n", (int)pass_criteria);
>  
>  	ret = 0;
> -	if (stats_max(&lock_wait_dat) > stats_min(&low_dat) + (int)pass_criteria)
> +	if (max_pi_delay > pass_criteria)
>  		ret = 1;
>  
>  	printf("Result: %s\n", ret ? "FAIL" : "PASS");
> 


------------------------------------------------------------------------------
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] [PATCH] realtime: Fix the pass criterion of pi_perf test case.

[Sripathi Kodi]

[-- Attachment #1: Type: text/plain, Size: 2304 bytes --]

On Monday 06 July 2009 21:20:49 Subrata Modak wrote:
> On Fri, 2009-07-03 at 12:35 +0530, Sripathi Kodi wrote:
> > Hi,
> >
> > The pass criterion in pi_perf test case is wrong. It compares the minimum
> > amount of time taken by the low priority thread with the maximum amount
> > of time taken by the high priority thread to calculate the PI delay.
> > Obviously, these min and max don't necessarily happen in the same
> > iteration, resulting in a number of false failures. The correct way is to
> > compare the time taken by low and high priority threads in each iteration
> > and then find the maximum delay experienced by high priority thread
> > across the iterations. This patch implements the change. Additionally,
> > this patch removes lock_wait_dat array, which is not needed anymore as
> > well as makes a couple of messages easier to understand.
> >
> > This patch changes the messages displayed by this test case slightly.
> > They look like the following now:
> >
> > Low prio thread started
> > High prio thread started
> > Busy 0 started
> > Busy 1 started
> > Busy 2 started
> > Busy 3 started
> > Time taken for high prio thread to get the lock once released by low prio
> > thread
> > Min delay = 10 us
> > Max delay = 36 us
> > Average delay = 17.06 us
> > Standard Deviation = 8.11 us
> > Quantiles:
> > 99.0% < 36
> > Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
> > Result: PASS
> >
> > I have tested this by running 1000 iterations of pi_perf test on a couple
> > of machines.
> >
> > Thanks,
> > Sripathi.
> >
> > ---
> >
> > realtime: Fix the pass criterion of pi_perf test case.
> >
> > Signed-off-by: Sripathi Kodi <sripathik@in.ibm.com>
>
> Hi Sri,
>
> Is this based on the latest snapshot. I get the following failures:
>
> patching file testcases/realtime/func/pi_perf/pi_perf.c
> Hunk #1 FAILED at 1.
>
>
> Can you please resend ?

Sorry Subrata, the patch seems to be fine on my machine, but is going bad in my 
mail client. If you don't mind, I have attached the patch to this mail for 
now.

Thanks,
Sripathi.

--

realtime: Fix the pass criterion of pi_perf test case.

Signed-off-by: Sripathi Kodi <sripathik@in.ibm.com>
Acked-by: Will Schmidt <will_schmidt@vnet.ibm.com>
Acked-by: Gowrishankar <gowrishankar.m@linux.vnet.ibm.com>

<snip>

[-- Attachment #2: pi_perf_criterion_fix_v2.patch --]
[-- Type: text/x-patch, Size: 3870 bytes --]

diff -uprN ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c
--- ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c	2009-07-03 12:21:41.000000000 +0530
+++ ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c	2009-07-03 12:24:44.000000000 +0530
@@ -1,6 +1,6 @@
 /******************************************************************************
  *
- *   Copyright © International Business Machines  Corp., 2007, 2008
+ *   Copyright © International Business Machines  Corp., 2007, 2008, 2009
  *
  *   This program is free software;  you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -36,6 +36,8 @@
  *
  * HISTORY
  *     2007-Nov-20:    Initial version by Sripathi Kodi <sripathik@in.ibm.com>
+ *     2009-Jul-03:    Pass criteria corrected by Sripathi Kodi
+ *                                                      <sripathik@in.ibm.com>
  *
  *****************************************************************************/
 
@@ -71,9 +73,9 @@ static unsigned int high_work_time = DEF
 static unsigned int busy_work_time;
 static int num_busy = -1;
 
-nsec_t low_unlock, high_get_lock;
+nsec_t low_unlock, max_pi_delay;
 
-stats_container_t lock_wait_dat, low_dat, cpu_delay_dat;
+stats_container_t low_dat, cpu_delay_dat;
 stats_container_t cpu_delay_hist;
 stats_quantiles_t cpu_delay_quantiles;
 
@@ -165,10 +167,9 @@ void * low_prio_thread(void *arg)
 
 void * high_prio_thread(void *arg)
 {
-	nsec_t high_start, high_spent, high_end;
+	nsec_t high_start, high_end, high_get_lock;
 	unsigned int i;
 
-	stats_container_init(&lock_wait_dat, iterations);
 	stats_container_init(&cpu_delay_dat, iterations);
 	stats_container_init(&cpu_delay_hist, HIST_BUCKETS);
 	stats_quantiles_init(&cpu_delay_quantiles, (int)log10(iterations));
@@ -184,14 +185,11 @@ void * high_prio_thread(void *arg)
 		high_start = rt_gettime();
 		pthread_mutex_lock(&lock);
 		high_end = rt_gettime();
-		high_spent = high_end - high_start;
 		high_get_lock = high_end - low_unlock;
 
 		busy_work_ms(high_work_time);
 		pthread_mutex_unlock(&lock);
 
-		lock_wait_dat.records[i].x = i;
-		lock_wait_dat.records[i].y = high_spent / NS_PER_US;
 		cpu_delay_dat.records[i].x = i;
 		cpu_delay_dat.records[i].y = high_get_lock / NS_PER_US;
 
@@ -206,14 +204,16 @@ void * high_prio_thread(void *arg)
 				"Latency (us)", "Samples", &cpu_delay_hist, "steps");
 
 	printf("Time taken for high prio thread to get the lock once released by low prio thread\n");
-	printf("Min wait time = %ld us\n", stats_min(&cpu_delay_dat));
-	printf("Max wait time = %ld us\n", stats_max(&cpu_delay_dat));
-	printf("Average wait time = %4.2f us\n", stats_avg(&cpu_delay_dat));
+	printf("Min delay = %ld us\n", stats_min(&cpu_delay_dat));
+	printf("Max delay = %ld us\n", stats_max(&cpu_delay_dat));
+	printf("Average delay = %4.2f us\n", stats_avg(&cpu_delay_dat));
 	printf("Standard Deviation = %4.2f us\n", stats_stddev(&cpu_delay_dat));
 	printf("Quantiles:\n");
 	stats_quantiles_calc(&cpu_delay_dat, &cpu_delay_quantiles);
 	stats_quantiles_print(&cpu_delay_quantiles);
 
+	max_pi_delay = stats_max(&cpu_delay_dat);
+
 	return NULL;
 }
 
@@ -261,13 +261,11 @@ int main(int argc, char *argv[])
 	}
 
 	join_threads();
-	printf("Low prio lock held time (min) = %ld us\n", stats_min(&low_dat));
-	printf("High prio lock wait time (max) = %ld us\n", stats_max(&lock_wait_dat));
 	printf("Criteria: High prio lock wait time < "
 			"(Low prio lock held time + %d us)\n", (int)pass_criteria);
 
 	ret = 0;
-	if (stats_max(&lock_wait_dat) > stats_min(&low_dat) + (int)pass_criteria)
+	if (max_pi_delay > pass_criteria)
 		ret = 1;
 
 	printf("Result: %s\n", ret ? "FAIL" : "PASS");

[-- Attachment #3: Type: text/plain, Size: 390 bytes --]

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have 
the opportunity to enter the BlackBerry Developer Challenge. See full prize 
details at: http://p.sf.net/sfu/blackberry

[-- Attachment #4: Type: text/plain, Size: 155 bytes --]

_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] [PATCH] realtime: Fix the pass criterion of pi_perf test case.

[Subrata Modak]

On Tue, 2009-07-07 at 12:11 +0530, Sripathi Kodi wrote: 
> On Monday 06 July 2009 21:20:49 Subrata Modak wrote:
> > On Fri, 2009-07-03 at 12:35 +0530, Sripathi Kodi wrote:
> > > Hi,
> > >
> > > The pass criterion in pi_perf test case is wrong. It compares the minimum
> > > amount of time taken by the low priority thread with the maximum amount
> > > of time taken by the high priority thread to calculate the PI delay.
> > > Obviously, these min and max don't necessarily happen in the same
> > > iteration, resulting in a number of false failures. The correct way is to
> > > compare the time taken by low and high priority threads in each iteration
> > > and then find the maximum delay experienced by high priority thread
> > > across the iterations. This patch implements the change. Additionally,
> > > this patch removes lock_wait_dat array, which is not needed anymore as
> > > well as makes a couple of messages easier to understand.
> > >
> > > This patch changes the messages displayed by this test case slightly.
> > > They look like the following now:
> > >
> > > Low prio thread started
> > > High prio thread started
> > > Busy 0 started
> > > Busy 1 started
> > > Busy 2 started
> > > Busy 3 started
> > > Time taken for high prio thread to get the lock once released by low prio
> > > thread
> > > Min delay = 10 us
> > > Max delay = 36 us
> > > Average delay = 17.06 us
> > > Standard Deviation = 8.11 us
> > > Quantiles:
> > > 99.0% < 36
> > > Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
> > > Result: PASS
> > >
> > > I have tested this by running 1000 iterations of pi_perf test on a couple
> > > of machines.
> > >
> > > Thanks,
> > > Sripathi.
> > >
> > > ---
> > >
> > > realtime: Fix the pass criterion of pi_perf test case.
> > >
> > > Signed-off-by: Sripathi Kodi <sripathik@in.ibm.com>
> >
> > Hi Sri,
> >
> > Is this based on the latest snapshot. I get the following failures:
> >
> > patching file testcases/realtime/func/pi_perf/pi_perf.c
> > Hunk #1 FAILED at 1.
> >
> >
> > Can you please resend ?
> 
> Sorry Subrata, the patch seems to be fine on my machine, but is going bad in my 
> mail client. If you don't mind, I have attached the patch to this mail for 
> now.
> 
> Thanks,
> Sripathi.

Thanks.

Regards--
Subrata

> 
> --
> 
> realtime: Fix the pass criterion of pi_perf test case.
> 
> Signed-off-by: Sripathi Kodi <sripathik@in.ibm.com>
> Acked-by: Will Schmidt <will_schmidt@vnet.ibm.com>
> Acked-by: Gowrishankar <gowrishankar.m@linux.vnet.ibm.com>
> 
> <snip>


------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have 
the opportunity to enter the BlackBerry Developer Challenge. See full prize 
details at: http://p.sf.net/sfu/blackberry
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list