Undefined behaviour of connect(fd, NULL, 0);

Undefined behaviour of connect(fd, NULL, 0);

[Neil Brown]

Hi Netdev.

We have a customer who was reporting strangely unpredictable behaviour of an
in-house application that used networking.

It called connect on a non-blocking socket and subsequently called
   connect(fd, NULL, 0)

to check if the connection had succeeded.
This would sometime "work" and sometimes close the connection.

Looking at the code (sys_connect, move_addr_to_kernel, inet_stream_connect),
it seems that in this case an uninitialised on-stack address is passed
to inet_stream_connect and it makes a decision based on ->sa_family (which is
uninitialised).

It seems clear that connect(fd, NULL, 0) is the wrong thing to do in this
circumstance, but I think it would be good if it failed consistently rather
than unpredictably.

Would it be appropriate for move_addr_to_kernel to zero out the remainder of
the address?
   memset(kaddr+ulen, 0, MAX_SOCK_ADDR-ulen);
??

Then connect(fd, NULL, 0) would always break the connection.

Thanks,
NeilBrown

Re: Undefined behaviour of connect(fd, NULL, 0);

[Stephen Hemminger]

On Wed, 31 Mar 2010 22:36:37 +1100
Neil Brown <neilb@suse.de> wrote:

> 
> Hi Netdev.
> 
> We have a customer who was reporting strangely unpredictable behaviour of an
> in-house application that used networking.
> 
> It called connect on a non-blocking socket and subsequently called
>    connect(fd, NULL, 0)
> 
> to check if the connection had succeeded.
> This would sometime "work" and sometimes close the connection.
> 
> Looking at the code (sys_connect, move_addr_to_kernel, inet_stream_connect),
> it seems that in this case an uninitialised on-stack address is passed
> to inet_stream_connect and it makes a decision based on ->sa_family (which is
> uninitialised).
> 
> It seems clear that connect(fd, NULL, 0) is the wrong thing to do in this
> circumstance, but I think it would be good if it failed consistently rather
> than unpredictably.
> 
> Would it be appropriate for move_addr_to_kernel to zero out the remainder of
> the address?
>    memset(kaddr+ulen, 0, MAX_SOCK_ADDR-ulen);
> ??
> 
> Then connect(fd, NULL, 0) would always break the connection.

I think the problem is inet_stream_connect referencing past addr_len.

--- a/net/ipv4/af_inet.c	2010-03-31 11:47:01.952910248 -0700
+++ b/net/ipv4/af_inet.c	2010-03-31 11:48:09.852938406 -0700
@@ -575,7 +575,7 @@ int inet_stream_connect(struct socket *s
 
 	lock_sock(sk);
 
-	if (uaddr->sa_family == AF_UNSPEC) {
+	if (addr_len < sizeof(sa_family_t) || uaddr->sa_family == AF_UNSPEC) {
 		err = sk->sk_prot->disconnect(sk, flags);
 		sock->state = err ? SS_DISCONNECTING : SS_UNCONNECTED;
 		goto out;

Re: Undefined behaviour of connect(fd, NULL, 0);

[Neil Brown]

On Wed, 31 Mar 2010 11:49:36 -0700
Stephen Hemminger <shemminger@vyatta.com> wrote:

> On Wed, 31 Mar 2010 22:36:37 +1100
> Neil Brown <neilb@suse.de> wrote:
> 
> > 
> > Hi Netdev.
> > 
> > We have a customer who was reporting strangely unpredictable behaviour of an
> > in-house application that used networking.
> > 
> > It called connect on a non-blocking socket and subsequently called
> >    connect(fd, NULL, 0)
> > 
> > to check if the connection had succeeded.
> > This would sometime "work" and sometimes close the connection.
> > 
> > Looking at the code (sys_connect, move_addr_to_kernel, inet_stream_connect),
> > it seems that in this case an uninitialised on-stack address is passed
> > to inet_stream_connect and it makes a decision based on ->sa_family (which is
> > uninitialised).
> > 
> > It seems clear that connect(fd, NULL, 0) is the wrong thing to do in this
> > circumstance, but I think it would be good if it failed consistently rather
> > than unpredictably.
> > 
> > Would it be appropriate for move_addr_to_kernel to zero out the remainder of
> > the address?
> >    memset(kaddr+ulen, 0, MAX_SOCK_ADDR-ulen);
> > ??
> > 
> > Then connect(fd, NULL, 0) would always break the connection.
> 
> I think the problem is inet_stream_connect referencing past addr_len.
> 
> --- a/net/ipv4/af_inet.c	2010-03-31 11:47:01.952910248 -0700
> +++ b/net/ipv4/af_inet.c	2010-03-31 11:48:09.852938406 -0700
> @@ -575,7 +575,7 @@ int inet_stream_connect(struct socket *s
>  
>  	lock_sock(sk);
>  
> -	if (uaddr->sa_family == AF_UNSPEC) {
> +	if (addr_len < sizeof(sa_family_t) || uaddr->sa_family == AF_UNSPEC) {
>  		err = sk->sk_prot->disconnect(sk, flags);
>  		sock->state = err ? SS_DISCONNECTING : SS_UNCONNECTED;
>  		goto out;

Thanks for the reply.

The implication of this patch is that
   connect(fd, NULL, 0)
is actually a valid way to check if an in-progress connection has completed.

Is that the intention?

Does all other address manipulation code check the addr_len ?? (probably).

Thanks,
NeilBrown

Re: Undefined behaviour of connect(fd, NULL, 0);

[Stephen Hemminger]

On Thu, 1 Apr 2010 07:24:12 +1100
Neil Brown <neilb@suse.de> wrote:

> On Wed, 31 Mar 2010 11:49:36 -0700
> Stephen Hemminger <shemminger@vyatta.com> wrote:
> 
> > On Wed, 31 Mar 2010 22:36:37 +1100
> > Neil Brown <neilb@suse.de> wrote:
> > 
> > > 
> > > Hi Netdev.
> > > 
> > > We have a customer who was reporting strangely unpredictable behaviour of an
> > > in-house application that used networking.
> > > 
> > > It called connect on a non-blocking socket and subsequently called
> > >    connect(fd, NULL, 0)
> > > 
> > > to check if the connection had succeeded.
> > > This would sometime "work" and sometimes close the connection.
> > > 
> > > Looking at the code (sys_connect, move_addr_to_kernel, inet_stream_connect),
> > > it seems that in this case an uninitialised on-stack address is passed
> > > to inet_stream_connect and it makes a decision based on ->sa_family (which is
> > > uninitialised).
> > > 
> > > It seems clear that connect(fd, NULL, 0) is the wrong thing to do in this
> > > circumstance, but I think it would be good if it failed consistently rather
> > > than unpredictably.
> > > 
> > > Would it be appropriate for move_addr_to_kernel to zero out the remainder of
> > > the address?
> > >    memset(kaddr+ulen, 0, MAX_SOCK_ADDR-ulen);
> > > ??
> > > 
> > > Then connect(fd, NULL, 0) would always break the connection.
> > 
> > I think the problem is inet_stream_connect referencing past addr_len.
> > 
> > --- a/net/ipv4/af_inet.c	2010-03-31 11:47:01.952910248 -0700
> > +++ b/net/ipv4/af_inet.c	2010-03-31 11:48:09.852938406 -0700
> > @@ -575,7 +575,7 @@ int inet_stream_connect(struct socket *s
> >  
> >  	lock_sock(sk);
> >  
> > -	if (uaddr->sa_family == AF_UNSPEC) {
> > +	if (addr_len < sizeof(sa_family_t) || uaddr->sa_family == AF_UNSPEC) {
> >  		err = sk->sk_prot->disconnect(sk, flags);
> >  		sock->state = err ? SS_DISCONNECTING : SS_UNCONNECTED;
> >  		goto out;
> 
> Thanks for the reply.
> 
> The implication of this patch is that
>    connect(fd, NULL, 0)
> is actually a valid way to check if an in-progress connection has completed.
> 
> Is that the intention?
The rationale is that move_addr_to_kernel, explcitly allow addr=NULL with addr_len=0
so if it is allowed there why not let it through. The implication of this is that
addr_len is the same as AF_UNSPEC.

> Does all other address manipulation code check the addr_len ?? (probably).

Not sure.
Someone ought to check BSD/Solaris to see if there is some standard here.

Re: Undefined behaviour of connect(fd, NULL, 0);

[David Miller]

From: Neil Brown <neilb@suse.de>
Date: Thu, 1 Apr 2010 07:24:12 +1100

>> --- a/net/ipv4/af_inet.c	2010-03-31 11:47:01.952910248 -0700
>> +++ b/net/ipv4/af_inet.c	2010-03-31 11:48:09.852938406 -0700
>> @@ -575,7 +575,7 @@ int inet_stream_connect(struct socket *s
>>  
>>  	lock_sock(sk);
>>  
>> -	if (uaddr->sa_family == AF_UNSPEC) {
>> +	if (addr_len < sizeof(sa_family_t) || uaddr->sa_family == AF_UNSPEC) {
>>  		err = sk->sk_prot->disconnect(sk, flags);
>>  		sock->state = err ? SS_DISCONNECTING : SS_UNCONNECTED;
>>  		goto out;
> 
> Thanks for the reply.
> 
> The implication of this patch is that
>    connect(fd, NULL, 0)
> is actually a valid way to check if an in-progress connection has completed.
> 
> Is that the intention?

That's not how I read the patch, the result is that connect(fd, NULL...)
will now disconnect the socket.

Re: Undefined behaviour of connect(fd, NULL, 0);

[Neil Brown]

On Wed, 31 Mar 2010 14:17:32 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:

> From: Neil Brown <neilb@suse.de>
> Date: Thu, 1 Apr 2010 07:24:12 +1100
> 
> >> --- a/net/ipv4/af_inet.c	2010-03-31 11:47:01.952910248 -0700
> >> +++ b/net/ipv4/af_inet.c	2010-03-31 11:48:09.852938406 -0700
> >> @@ -575,7 +575,7 @@ int inet_stream_connect(struct socket *s
> >>  
> >>  	lock_sock(sk);
> >>  
> >> -	if (uaddr->sa_family == AF_UNSPEC) {
> >> +	if (addr_len < sizeof(sa_family_t) || uaddr->sa_family == AF_UNSPEC) {
> >>  		err = sk->sk_prot->disconnect(sk, flags);
> >>  		sock->state = err ? SS_DISCONNECTING : SS_UNCONNECTED;
> >>  		goto out;
> > 
> > Thanks for the reply.
> > 
> > The implication of this patch is that
> >    connect(fd, NULL, 0)
> > is actually a valid way to check if an in-progress connection has completed.
> > 
> > Is that the intention?
> 
> That's not how I read the patch, the result is that connect(fd, NULL...)
> will now disconnect the socket.

Yes, you are right - I read it upside-down.  Sorry.

Thanks,
NeilBrown

Re: Undefined behaviour of connect(fd, NULL, 0);

[Changli Gao]

I think the following patch is what Neil wants. The old code implies that
connect(fd, NULL, 0) is used to check the socket connecting status, but
Stephen's patch breaks it. The old code is wrong when it checks the address's
faimly but not check the sizeof of the address to determine the family member
is valid or not before.

diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index be1a6ac..3ff51f0 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -576,7 +576,8 @@ int inet_stream_connect(struct socket *sock,
struct sockaddr *uaddr,

 	lock_sock(sk);

-	if (uaddr->sa_family == AF_UNSPEC) {
+	if (addr_len >= sizeof(uaddr->sa_family) &&
+	    uaddr->sa_family == AF_UNSPEC) {
 		err = sk->sk_prot->disconnect(sk, flags);
 		sock->state = err ? SS_DISCONNECTING : SS_UNCONNECTED;
 		goto out;

Re: Undefined behaviour of connect(fd, NULL, 0);

[Neil Brown]

On Thu, 1 Apr 2010 11:00:23 +0800
Changli Gao <xiaosuo@gmail.com> wrote:

> I think the following patch is what Neil wants. The old code implies that
> connect(fd, NULL, 0) is used to check the socket connecting status, but
> Stephen's patch breaks it. The old code is wrong when it checks the address's
> faimly but not check the sizeof of the address to determine the family member
> is valid or not before.
> 
> diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
> index be1a6ac..3ff51f0 100644
> --- a/net/ipv4/af_inet.c
> +++ b/net/ipv4/af_inet.c
> @@ -576,7 +576,8 @@ int inet_stream_connect(struct socket *sock,
> struct sockaddr *uaddr,
> 
>  	lock_sock(sk);
> 
> -	if (uaddr->sa_family == AF_UNSPEC) {
> +	if (addr_len >= sizeof(uaddr->sa_family) &&
> +	    uaddr->sa_family == AF_UNSPEC) {
>  		err = sk->sk_prot->disconnect(sk, flags);
>  		sock->state = err ? SS_DISCONNECTING : SS_UNCONNECTED;
>  		goto out;


I'm not sure I'd say that I "want" any particular patch.
I just want to know what "connect(fd, NULL, 0)" is supposed to do, and to
have the kernel be consistent in its behaviour.  I'm not really fussed what
the behaviour is.

I suspect the customer wants that patch you have supplied as it would mean
they don't need to change their code.  But I only want it if it is "right".

The patch you have provided does what I had assumed Stephen's patch did
before I actually read it properly.

My feeling is that this patch might be more useful than Stephen's as having
connect(fd, NULL, 0) do what the customer expects seems useful, where as
having it do the same as setting AF_UNSPEC doesn't add anything.

I've googled around a bit but cannot find any evidence of anyone passing NULL
to connect like this, and what documentation I can find doesn't really
address the issue at all.

Thanks,
NeilBrown

Re: Undefined behaviour of connect(fd, NULL, 0);

[Changli Gao]

On Thu, Apr 1, 2010 at 11:38 AM, Neil Brown <neilb@suse.de> wrote:
> I'm not sure I'd say that I "want" any particular patch.
> I just want to know what "connect(fd, NULL, 0)" is supposed to do, and to
> have the kernel be consistent in its behaviour.  I'm not really fussed what
> the behaviour is.
>
> I suspect the customer wants that patch you have supplied as it would mean
> they don't need to change their code.  But I only want it if it is "right".
>
> The patch you have provided does what I had assumed Stephen's patch did
> before I actually read it properly.
>
> My feeling is that this patch might be more useful than Stephen's as having
> connect(fd, NULL, 0) do what the customer expects seems useful, where as
> having it do the same as setting AF_UNSPEC doesn't add anything.
>
> I've googled around a bit but cannot find any evidence of anyone passing NULL
> to connect like this, and what documentation I can find doesn't really
> address the issue at all.
>

I found this from man page for connect(2)

       Generally, connection-based protocol sockets may successfully connect()
       only once; connectionless protocol sockets may use  connect()  multiple
       times to change their association.  Connectionless sockets may dissolve
       the association by connecting to an address with the  sa_family  member
       of sockaddr set to AF_UNSPEC (supported on Linux since kernel 2.2).

It only said the meaning of AF_UNSPEC for connectionless sockets, but
not connection sockets like TCP. It means that the behavior of
disconnecting the socket when AF_UNSEPC family address is met is also
undocumented.

The connect() API is a little different. If you try to call connect()
in non-blocking mode, and the API can't connect instantly, it will
return the error code for 'Operation In Progress'. When you call
connect() again, later, it may tell you 'Operation Already In
Progress' to let you know that it's still trying to connect, or it may
give you a successful return code, telling you that the connect has
been made.
from: http://www.scottklement.com/rpg/socktut/nonblocking.html

Someone may use connect() to check if the connection is established or
not. But there is no spec about the addr and addr_len value when
connect(2) is used  this way. Since there is no limit of addr and
addr_len, and we supports addr is NULL to check the status of socket
(Although it is buggy). I think we should treat it like a feature, and
the problem Neil reported is a bug.

-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: Undefined behaviour of connect(fd, NULL, 0);

[Changli Gao]

On Thu, Apr 1, 2010 at 12:16 PM, Changli Gao <xiaosuo@gmail.com> wrote:
>
> I found this from man page for connect(2)
>
>       Generally, connection-based protocol sockets may successfully connect()
>       only once; connectionless protocol sockets may use  connect()  multiple
>       times to change their association.  Connectionless sockets may dissolve
>       the association by connecting to an address with the  sa_family  member
>       of sockaddr set to AF_UNSPEC (supported on Linux since kernel 2.2).
>

dissolving the association by connecting to an address with the
sa_family member of sockaddr set to AF_UNSEPC is broken too.

int ip4_datagram_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
{
        struct inet_sock *inet = inet_sk(sk);
        struct sockaddr_in *usin = (struct sockaddr_in *) uaddr;
        struct rtable *rt;
        __be32 saddr;
        int oif;
        int err;


        if (addr_len < sizeof(*usin))
                return -EINVAL;

        if (usin->sin_family != AF_INET)
                return -EAFNOSUPPORT;

according to the man page, sin_family == AF_UNSPEC should be allowed.
And netlink's connect doesn't check the addr_len, so it behavior is
also undeterminedl

static int netlink_connect(struct socket *sock, struct sockaddr *addr,
                           int alen, int flags)
{
        int err = 0;
        struct sock *sk = sock->sk;
        struct netlink_sock *nlk = nlk_sk(sk);
        struct sockaddr_nl *nladdr = (struct sockaddr_nl *)addr;

        if (addr->sa_family == AF_UNSPEC) {
                sk->sk_state    = NETLINK_UNCONNECTED;
                nlk->dst_pid    = 0;
                nlk->dst_group  = 0;
                return 0;
        }

If this issues need to be fixed, I'll check all the protocols if their
connect() checkes the sizeof of socket address or not, and post a
patch.

-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: Undefined behaviour of connect(fd, NULL, 0);

[David Miller]

From: Changli Gao <xiaosuo@gmail.com>
Date: Thu, 1 Apr 2010 12:16:43 +0800

> Someone may use connect() to check if the connection is established
> or not. But there is no spec about the addr and addr_len value when
> connect(2) is used this way. Since there is no limit of addr and
> addr_len, and we supports addr is NULL to check the status of socket
> (Although it is buggy). I think we should treat it like a feature,
> and the problem Neil reported is a bug.

This seems logical, but I believe it is wrong.

We already know for a fact that it is guarenteed to not work
reliably for every single kernel in existence in the world
right now.

Every system.  Ones that have been deployed for 10 years as
well as those built from GIT 10 seconds ago.

So you tell me, if you put this into an application that you
wish to deploy anywhere, are you not being completely stupid?

Therefore, if it's illogical to use this in an application, what value
is there in starting to support it now in the kernel?

I'll tell you, the value is absolutely zero.

Yes we need to add the length check, but the behavior we give to this
case as a result, is completely arbitrary.  And I would in fact argue
for a hard error in these cases.

Simply mark it as invalid to call connect() this way.

Re: Undefined behaviour of connect(fd, NULL, 0);

[Changli Gao]

On Thu, Apr 1, 2010 at 3:23 PM, David Miller <davem@davemloft.net> wrote:
>
> This seems logical, but I believe it is wrong.
>
> We already know for a fact that it is guarenteed to not work
> reliably for every single kernel in existence in the world
> right now.
>
> Every system.  Ones that have been deployed for 10 years as
> well as those built from GIT 10 seconds ago.
>
> So you tell me, if you put this into an application that you
> wish to deploy anywhere, are you not being completely stupid?
>
> Therefore, if it's illogical to use this in an application, what value
> is there in starting to support it now in the kernel?
>
> I'll tell you, the value is absolutely zero.
>
> Yes we need to add the length check, but the behavior we give to this
> case as a result, is completely arbitrary.  And I would in fact argue
> for a hard error in these cases.
>
> Simply mark it as invalid to call connect() this way.
>

I found this from the man page of FreeBSD's connect(2).

     Generally, stream sockets may successfully connect() only
     once; datagram sockets may use connect() multiple times to change their
     association.  Datagram sockets may dissolve the association by connecting
     to an invalid address, such as a null address.

And this from the man page of Darwin's connect(2).

     Datagram sockets may dissolve the association by connecting to an
     invalid address, such as a null address or an address with the address
     family set to AF_UNSPEC (the error EAFNOSUPPORT will be harmlessly
     returned).

Since null address behavior has been defined by the others. I think
Linux should be compatible with the others. So the  patch submitted on
this by me should not been applied. I'll work out another patch later.

-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

Re: Undefined behaviour of connect(fd, NULL, 0);

[Eric Dumazet]

Le lundi 05 avril 2010 à 17:23 +0800, Changli Gao a écrit :

> I found this from the man page of FreeBSD's connect(2).
> 
>      Generally, stream sockets may successfully connect() only
>      once; datagram sockets may use connect() multiple times to change their
>      association.  Datagram sockets may dissolve the association by connecting
>      to an invalid address, such as a null address.
> 
> And this from the man page of Darwin's connect(2).
> 
>      Datagram sockets may dissolve the association by connecting to an
>      invalid address, such as a null address or an address with the address
>      family set to AF_UNSPEC (the error EAFNOSUPPORT will be harmlessly
>      returned).
> 
> Since null address behavior has been defined by the others. I think
> Linux should be compatible with the others. So the  patch submitted on
> this by me should not been applied. I'll work out another patch later.
> 

As pointed by David, no sane application would use this facility until a
decade, I wonder why you insist so much for this minor detail.

Solaris man page extract :

"Datagram sockets can dissolve the association by connecting to a null
address."

What is a null address ? 

1) A null pointer ? 
2) a pointer to a zone, but length of this zone is 0
3) Or a pointer to a zone filled with NULL bytes ?

Linux implements the later interpretation. Its more than enough.

If a NULL pointer was implemented, man pages would use the following
words : "Datagram sockets can dissolve the association by connecting to
a NULL pointer (NULL second argument to connect())."

If you submit a patch to change connect() behavior, dont forget to send
appropriate changes to Michael, because in the end, nobody but you knows
how things are supposed to work if not documented.

MAN-PAGES: MANUAL PAGES FOR LINUX -- Sections 2, 3, 4, 5, and 7
M:      Michael Kerrisk <mtk.manpages@gmail.com>
W:      http://www.kernel.org/doc/man-pages

Re: Undefined behaviour of connect(fd, NULL, 0);

[Andreas Schwab]

Eric Dumazet <eric.dumazet@gmail.com> writes:

> Solaris man page extract :
>
> "Datagram sockets can dissolve the association by connecting to a null
> address."
>
> What is a null address ? 
>
> 1) A null pointer ? 
> 2) a pointer to a zone, but length of this zone is 0
> 3) Or a pointer to a zone filled with NULL bytes ?

Btw., POSIX.1 has changed the description from "If address is a null
address for the protocol, the socket's peer address shall be reset" in
the 2004 edition to "If the sa_family member of address is AF_UNSPEC,
the socket's peer address shall be reset" in the 2009 edition.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Undefined behaviour of connect(fd, NULL, 0);

[David Miller]

From: Changli Gao <xiaosuo@gmail.com>
Date: Mon, 5 Apr 2010 17:23:50 +0800

> I found this from the man page of FreeBSD's connect(2).
> 
>      Generally, stream sockets may successfully connect() only
>      once; datagram sockets may use connect() multiple times to change their
>      association.  Datagram sockets may dissolve the association by connecting
>      to an invalid address, such as a null address.
> 
> And this from the man page of Darwin's connect(2).
> 
>      Datagram sockets may dissolve the association by connecting to an
>      invalid address, such as a null address or an address with the address
>      family set to AF_UNSPEC (the error EAFNOSUPPORT will be harmlessly
>      returned).
> 
> Since null address behavior has been defined by the others. I think
> Linux should be compatible with the others. So the  patch submitted on
> this by me should not been applied. I'll work out another patch later.

Your patch is already applied and pushed out to the world.
Any corrections would need to be made relative to that.

But I think no changes should be made, and it doesn't matter
at all what some manual page says on another system.

It doesn't matter one iota what those paragraphs say, we've had the
existing behavior for FIFTEEN YEARS.  So, like I said last time, any
application coding to this behavior of the zero length sockaddr doing
anything on connect() will NOT WORK ON ANY KERNEL.

So supporting it has absolutely zero value, for us and for app
developers.