Question on HDLC and raw access to T1/E1 serial streams.

Question on HDLC and raw access to T1/E1 serial streams.

[Ben Greear]

I am looking for a way to bridge a T1/E1 network by reading a raw 
bitstream from one T1 interface and writing it out to the other.  The 
application is adding delay and/or bit corruptions for impairment testing.

I have been using Sangoma's drivers and NICs, but I'm having no luck 
getting their latest stuff to work so I was hoping to use in-kernel
drivers (even if that means writing or hiring someone to write new ones.)

Is there currently a way to read/write the raw bitstream for a full T1 
or E1 or a subset of channels?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Krzysztof Halasa]

Ben Greear <greearb@candelatech.com> writes:

> I am looking for a way to bridge a T1/E1 network by reading a raw
> bitstream from one T1 interface and writing it out to the other.  The
> application is adding delay and/or bit corruptions for impairment
> testing.
>
> I have been using Sangoma's drivers and NICs, but I'm having no luck
> getting their latest stuff to work so I was hoping to use in-kernel
> drivers (even if that means writing or hiring someone to write new ones.)
>
> Is there currently a way to read/write the raw bitstream for a full T1
> or E1 or a subset of channels?

Well, my generic HDLC works with HDLC framing only, T1/E1 is
a layer lower than that... I think Cyclades have (had?) a version
of PC300 card with T1/E1 interface. It at least doesn't require
any "binary blobs", though I think the driver would need some work.

Which line interface do you need? G.703?
Do you need to bridge multiple streams (not slots) over one
interface (internal (de)multiplexer - I mean "more than one
subset of channels")?
-- 
Krzysztof Halasa

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Ben Greear]

Krzysztof Halasa wrote:
> Ben Greear <greearb@candelatech.com> writes:
>
>   
>> I am looking for a way to bridge a T1/E1 network by reading a raw
>> bitstream from one T1 interface and writing it out to the other.  The
>> application is adding delay and/or bit corruptions for impairment
>> testing.
>>
>> I have been using Sangoma's drivers and NICs, but I'm having no luck
>> getting their latest stuff to work so I was hoping to use in-kernel
>> drivers (even if that means writing or hiring someone to write new ones.)
>>
>> Is there currently a way to read/write the raw bitstream for a full T1
>> or E1 or a subset of channels?
>>     
>
> Well, my generic HDLC works with HDLC framing only, T1/E1 is
> a layer lower than that... I think Cyclades have (had?) a version
> of PC300 card with T1/E1 interface. It at least doesn't require
> any "binary blobs", though I think the driver would need some work.
>
> Which line interface do you need? G.703?
> Do you need to bridge multiple streams (not slots) over one
> interface (internal (de)multiplexer - I mean "more than one
> subset of channels")?
>   
For protocols running HDLC as transport, I can just bridge HDLC frames.  
I would want
to be able to select the channel(s) for the HDLC frames.

For bridging something like voice, I think if I could break out an 
individual channel into
a bit-stream interface, I could just read bits off on one T1 channel and 
write to the other.
Preferably, I could also bond multiple channels (including an entire T1 
or E1) and bridge
it as raw bits too.

I think if I could support these scenarios below, I would have 
everything I need:

*  Configure T1 as unchannelized bitstream, bridge entire thing to 
second T1.

*  Configure channels 1-5 as a bitstream and bridge that to channels 1-5 
of a second T1.  (random proprietary bit-streaming protocol,
                     would probably bridge HDLC just fine, but handling 
HDLC as frames would be more efficient I think.)
    channels 6-10 configured as an HDLC interface, bridged as HDLC 
frames to channels 6-10 of a second T1. (PPP & other protocols over HDLC)
    channels 10-24 each configured as a separate bit-stream, bridged to 
channels 10-24 on the second T1. (Voice)

*  Configure entire T1 as HDLC transport, bridge HDLC frames from one T1 
to the other.

Does that make sense?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Krzysztof Halasa]

Ben Greear <greearb@candelatech.com> writes:

> I think if I could support these scenarios below, I would have
> everything I need:
>
> *  Configure T1 as unchannelized bitstream, bridge entire thing to
> second T1.

I think it should be easy with such card, though I think the drivers
can't currently do that.

> *  Configure channels 1-5 as a bitstream and bridge that to channels
> 1-5
> of a second T1.  (random proprietary bit-streaming protocol,

I think the hardware would permit that. Probably needs additional
driver support and I'm not sure about timeslot synchronization (for
HDLC, sync doesn't matter).

>                     would probably bridge HDLC just fine, but handling
> HDLC as frames would be more efficient I think.)

Not sure, maybe yes (less data to bridge due to bit stuffing, flags etc.),
maybe not (variable length of HDLC frame).

>    channels 6-10 configured as an HDLC interface, bridged as HDLC
> frames to channels 6-10 of a second T1. (PPP & other protocols over
> HDLC)
>    channels 10-24 each configured as a separate bit-stream, bridged to
> channels 10-24 on the second T1. (Voice)

I think the above could be a problem - I think common T1/E1 cards
can do only one stream at once.

I wonder if it can be done in software - the hardware framer would
have to pass all data transparently, and it would be demultiplexed,
processed and then multiplexed by the driver. Quite complicated,
but I think at 2 Mbps it wouldn't be a CPU performance problem.

> *  Configure entire T1 as HDLC transport, bridge HDLC frames from one
> T1 to the other.

Easy even with existing drivers I think (no bridge support but it's
trivial).
-- 
Krzysztof Halasa

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Ben Greear]

Krzysztof Halasa wrote:
> Ben Greear <greearb@candelatech.com> writes:
>
>   
>> I think if I could support these scenarios below, I would have
>> everything I need:
>>
>> *  Configure T1 as unchannelized bitstream, bridge entire thing to
>> second T1.
>>     
>
> I think it should be easy with such card, though I think the drivers
> can't currently do that.
>   
At least some out-of-tree drivers seem to be able to do this.  For 
instance, these guys:
http://www.farsite.co.uk/

>> *  Configure channels 1-5 as a bitstream and bridge that to channels
>> 1-5
>> of a second T1.  (random proprietary bit-streaming protocol,
>>     
>
> I think the hardware would permit that. Probably needs additional
> driver support and I'm not sure about timeslot synchronization (for
> HDLC, sync doesn't matter).
>   
My assumption for bridging a bitstream is that timeslot sync is not 
absolutely critical.  However, IF
you could be sure of time-slot sync, you'd  have a lot more power and be 
able to do some extra ticks
in user-space I think...
>>                     would probably bridge HDLC just fine, but handling
>> HDLC as frames would be more efficient I think.)
>>     
>
> Not sure, maybe yes (less data to bridge due to bit stuffing, flags etc.),
> maybe not (variable length of HDLC frame).
>   
The key for me is that if you ever miss a slot in bit-stream mode, you 
can never make it up because
every bit is critical.  This leads to having to drop arbitrary data to 
keep from ever-increasing latency on your
bridge.  With HDLC, you can skip the flags and make up time if you ever 
miss a timeslot (assuming the
HDLC is not using the line at 100% capacity.)
>>    channels 6-10 configured as an HDLC interface, bridged as HDLC
>> frames to channels 6-10 of a second T1. (PPP & other protocols over
>> HDLC)
>>    channels 10-24 each configured as a separate bit-stream, bridged to
>> channels 10-24 on the second T1. (Voice)
>>     
>
> I think the above could be a problem - I think common T1/E1 cards
> can do only one stream at once.
>
> I wonder if it can be done in software - the hardware framer would
> have to pass all data transparently, and it would be demultiplexed,
> processed and then multiplexed by the driver. Quite complicated,
> but I think at 2 Mbps it wouldn't be a CPU performance problem.
>   
I'd be happy with a software approach.  In fact, if I could get a framed 
packet (ie, I know that
byte 0 is channel 1, byte 24 is channel 24, and byte 25 is channel 1 
again...) then I could even
do the multiplexing in user space.

For write, I'd also need to be able to guarantee that byte 0 goes to 
channel 1, etc.  So, if the
driver bit-stuffed, then it would need to do an entire time-slice at once.

>> *  Configure entire T1 as HDLC transport, bridge HDLC frames from one
>> T1 to the other.
>>     
>
> Easy even with existing drivers I think (no bridge support but it's
> trivial).
>   
Excellent.  I actually want to write the bridge logic myself in 
user-space..I just need the driver
API and at least one driver that supports it and has support for readily 
available T1/E1 hardware.
For future work, I'd be interested in the same sort of support for DS3/E3.

I'd be willing to fund development of these features if you or someone 
else is interested
(please contact me off list for details.)

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Alan Cox]

Ar Sul, 2006-10-01 am 11:41 -0700, ysgrifennodd Ben Greear:
> At least some out-of-tree drivers seem to be able to do this.  For 
> instance, these guys:
> http://www.farsite.co.uk/


The crazy Zaptel guys can probably do precisely what you need as their
hardware though aimed at voice work lets software get in at an extremely
low level.

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Ben Greear]

Alan Cox wrote:
> Ar Sul, 2006-10-01 am 11:41 -0700, ysgrifennodd Ben Greear:
>   
>> At least some out-of-tree drivers seem to be able to do this.  For 
>> instance, these guys:
>> http://www.farsite.co.uk/
>>     
>
>
> The crazy Zaptel guys can probably do precisely what you need as their
> hardware though aimed at voice work lets software get in at an extremely
> low level.
>   
I think you might be right.  It will take some more experimenting to see 
if they allow
enough control to read/write bitstreams (clear channel groups in their 
terminology, I believe) while at
the same time keeping the timing slots in order...

Thanks,
Ben

> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>   


-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Lennart Sorensen]

On Tue, Oct 03, 2006 at 09:27:58PM -0700, Ben Greear wrote:
> I think you might be right.  It will take some more experimenting to see 
> if they allow
> enough control to read/write bitstreams (clear channel groups in their 
> terminology, I believe) while at
> the same time keeping the timing slots in order...

Sangoma also makes cards with linux drivers, which I suspect will let
you at the low levels if you want to.  They certainly support zaptel
stuff on their cards.  Their hardware is also quite a bit cheaper than
farsite's in my experience.

--
Len Sorensen

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Ben Greear]

Lennart Sorensen wrote:
> On Tue, Oct 03, 2006 at 09:27:58PM -0700, Ben Greear wrote:
>   
>> I think you might be right.  It will take some more experimenting to see 
>> if they allow
>> enough control to read/write bitstreams (clear channel groups in their 
>> terminology, I believe) while at
>> the same time keeping the timing slots in order...
>>     
>
> Sangoma also makes cards with linux drivers, which I suspect will let
> you at the low levels if you want to.  They certainly support zaptel
> stuff on their cards.  Their hardware is also quite a bit cheaper than
> farsite's in my experience.
>   
Sangoma's NICs used to work for me..but it seems their latest drivers 
broke the
support that I need.  I think there aren't too many people interested in 
bit-streaming, so
the feature probably doesn't get tested that often...  If I can get the 
digium hardware to
work with a particular zaptel.conf, then I can try Sangoma and see if it 
will work when
configured in that same manner...

Thanks,
Ben

> --
> Len Sorensen
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>   


-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Krzysztof Halasa]

Ben Greear <greearb@candelatech.com> writes:

> My assumption for bridging a bitstream is that timeslot sync is not
> absolutely critical.  However, IF
> you could be sure of time-slot sync, you'd  have a lot more power and
> be able to do some extra ticks
> in user-space I think...

Not sure what do you want to do with that, but it may be critical for
many things.

> The key for me is that if you ever miss a slot in bit-stream mode, you
> can never make it up because
> every bit is critical.

I think you'd have to perform some recover procedure then, that's similar
to DCE losing sync.

>  This leads to having to drop arbitrary data to
> keep from ever-increasing latency on your
> bridge.

If your clocks are synchronized (for example, if you get a "master"
clock from your public phone exchange and you propagate it downstream,
or your machine is the "master") then you never drop anything, the
input and output rates are equal and in sync.

>  With HDLC, you can skip the flags and make up time if you
> ever miss a timeslot (assuming the
> HDLC is not using the line at 100% capacity.)

Sure. Even at 100% you can just drop a frame, HDLC applications must
be prepared for it.

> I'd be happy with a software approach.  In fact, if I could get a
> framed packet (ie, I know that
> byte 0 is channel 1, byte 24 is channel 24, and byte 25 is channel 1
> again...) then I could even
> do the multiplexing in user space.

Well, with software framing it would look a bit different - there
is no such thing as "packet" as both TX and RX is continuous, not
aligned to anything etc. You would have to detect channel boundaries,
and bit-shift the data. Requires the sync serial controller (and FALC)
in transparent mode (I would have to look at some docs). I think
the kernel is a better place for things like that due to latency
issues.

> For write, I'd also need to be able to guarantee that byte 0 goes to
> channel 1, etc.  So, if the
> driver bit-stuffed, then it would need to do an entire time-slice at once.

BTW: An HDLC frame can use many slices. You can in fact have many
HDLC frames (from different streams) multiplexed. You just need
a multi-stream device or a multiplexer.

>>> *  Configure entire T1 as HDLC transport, bridge HDLC frames from one
>>> T1 to the other.
> Excellent.  I actually want to write the bridge logic myself in
> user-space..I just need the driver
> API and at least one driver that supports it and has support for
> readily available T1/E1 hardware.

If you want the userspace HDLC bridge... I'd use a pair of T1/E1 cards
with generic HDLC support, for example, Cyclades PC300 (never used them
and while I don't exactly like their driver, in case of problems I could
add T1/E1 to my own driver which currently supports PC300 X.21 and
V.24/V.35).

Once T1/E1s are working and the required slots are selected:
sethdlc hdlc0 hdlc (options)
sethdlc hdlc1 hdlc (options)
ifconfig hdlc0 up
ifconfig hdlc1 up
man PF_PACKET

A single HDLC stream is a simple thing because it's exactly what
the cards are designed for.
-- 
Krzysztof Halasa

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Ben Greear]

Krzysztof Halasa wrote:
> Ben Greear <greearb@candelatech.com> writes:
>
>   
>> My assumption for bridging a bitstream is that timeslot sync is not
>> absolutely critical.  However, IF
>> you could be sure of time-slot sync, you'd  have a lot more power and
>> be able to do some extra ticks
>> in user-space I think...
>>     
>
> Not sure what do you want to do with that, but it may be critical for
> many things.
>
>   
>> The key for me is that if you ever miss a slot in bit-stream mode, you
>> can never make it up because
>> every bit is critical.
>>     
>
> I think you'd have to perform some recover procedure then, that's similar
> to DCE losing sync.
>   
The recovery is to simply drop a few time slices.  Whatever protocol is 
running over
the wire must be able to deal with it.
>>  This leads to having to drop arbitrary data to
>> keep from ever-increasing latency on your
>> bridge.
>>     
>
> If your clocks are synchronized (for example, if you get a "master"
> clock from your public phone exchange and you propagate it downstream,
> or your machine is the "master") then you never drop anything, the
> input and output rates are equal and in sync.
>   

If you are bridging in user-space (or kernel, for that matter), it is 
possible your writing process
will not be scheduled in time to fill up the tx buffer with real data.  
So, it would have to be
padded by the driver and/or NIC hardware.  With an adequately powered 
machine, this should
be a rare occurrence, however.

>>  With HDLC, you can skip the flags and make up time if you
>> ever miss a timeslot (assuming the
>> HDLC is not using the line at 100% capacity.)
>>     
>
> Sure. Even at 100% you can just drop a frame, HDLC applications must
> be prepared for it.
>   
Agreed.
>> I'd be happy with a software approach.  In fact, if I could get a
>> framed packet (ie, I know that
>> byte 0 is channel 1, byte 24 is channel 24, and byte 25 is channel 1
>> again...) then I could even
>> do the multiplexing in user space.
>>     
>
> Well, with software framing it would look a bit different - there
> is no such thing as "packet" as both TX and RX is continuous, not
> aligned to anything etc. You would have to detect channel boundaries,
> and bit-shift the data. Requires the sync serial controller (and FALC)
> in transparent mode (I would have to look at some docs). I think
> the kernel is a better place for things like that due to latency
> issues.
>   
I would definitely want the driver/hardware to deal with the byte/bit 
boundaries.  But,
the user-space read API could be similar to UDP (or HDLC if I understand 
it correctly),
but it would always return a multiple of the number of channels in bytes 
(0, 24, 48, ... for and entire T1).
>> For write, I'd also need to be able to guarantee that byte 0 goes to
>> channel 1, etc.  So, if the
>> driver bit-stuffed, then it would need to do an entire time-slice at once.
>>     
>
> BTW: An HDLC frame can use many slices. You can in fact have many
> HDLC frames (from different streams) multiplexed. You just need
> a multi-stream device or a multiplexer.
>   
Good.
>>>> *  Configure entire T1 as HDLC transport, bridge HDLC frames from one
>>>> T1 to the other.
>>>>         
>> Excellent.  I actually want to write the bridge logic myself in
>> user-space..I just need the driver
>> API and at least one driver that supports it and has support for
>> readily available T1/E1 hardware.
>>     
>
> If you want the userspace HDLC bridge... I'd use a pair of T1/E1 cards
> with generic HDLC support, for example, Cyclades PC300 (never used them
> and while I don't exactly like their driver, in case of problems I could
> add T1/E1 to my own driver which currently supports PC300 X.21 and
> V.24/V.35).
>
> Once T1/E1s are working and the required slots are selected:
> sethdlc hdlc0 hdlc (options)
> sethdlc hdlc1 hdlc (options)
> ifconfig hdlc0 up
> ifconfig hdlc1 up
> man PF_PACKET
>
> A single HDLC stream is a simple thing because it's exactly what
> the cards are designed for.
>   
Before I settle on hardware, I'd like to have some idea that the NIC can 
deal with raw
bit-streams.  It seems all of the NICs can handle HDLC, so that part 
should be pretty
easy.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com> 
Candela Technologies Inc  http://www.candelatech.com

Re: Question on HDLC and raw access to T1/E1 serial streams.

[Krzysztof Halasa]

Ben Greear <greearb@candelatech.com> writes:

> I would definitely want the driver/hardware to deal with the byte/bit
> boundaries.  But,
> the user-space read API could be similar to UDP (or HDLC if I
> understand it correctly),

I think so.

> Before I settle on hardware, I'd like to have some idea that the NIC
> can deal with raw
> bit-streams.

I will look at FALC docs. Most (?) sync serial processors can RX/TX
transparent data (though for E3 rates it would be better to have
some hardware support).
-- 
Krzysztof Halasa