What are the ways of communication B/W User Space and Kernel Space

What is User Space and Kernel Space:

              Operating system segregates virtual memory into kernel space and user space. 

Kernel space is strictly reserved for running the kernel, kernel extensions, and most device drivers.

In contrast, user space is the memory area where all user mode applications work and this memory can be swapped out when necessary.

Different ways of Communication b/w User Space and Kernel Space:

There are many ways to Communicate between the User space and Kernel Space, they are:

File system based communication:

• Procfs 
• Sysfs 
• Configfs 
• Debugfs 
• Sysctl
• Character Devices 

Socket Based Communication: 

There are two types of sockets used for communication.

UDP Sockets

Netlink Socket.

In this post, i will be giving more importance for the Netlink socket based communication since this mechanism is extensively used in Linux networking applications.

What is NETLINK Socket:   

• Netlink socket is a special IPC used for transferring information between kernel and user-space processes.    

• It provides a full-duplex communication link between the two by way of standard socket APIs for user-space processes and a special kernel API for kernel modules.

• vNetlink socket uses the address family AF_NETLINK, as compared to AF_INET used by TCP/IP socket.
• vEach netlink socket feature defines its own protocol type in the kernel header file include/linux/netlink.h 

 Why do the above features use netlink instead of system calls, ioctls or proc filesystems for communication between user and kernel worlds?

• vIt is a nontrivial task to add system calls, ioctls or proc files for new features; we risk polluting the kernel and damaging the stability of the system.
• vNetlink socket is simple, though: only a constant, the protocol type, needs to be added to netlink.h.
• vThen, the kernel module and application can talk using socket-style APIs immediately
• vNetlink is asynchronous because, as with any other socket API, it provides a socket queue to smooth the burst of messages.
• vThe system call for sending a netlink message queues the message to the receiver's netlink queue and then invokes the receiver's reception handler.
• vThe receiver, within the reception handler's context, can decide whether to process the message immediately or leave the message in the queue and process it later in a different context.
• vUnlike netlink, system calls require synchronous processing. Therefore, if we use a system call to pass a message from user space to the kernel, the kernel scheduling granularity may be affected if the time to process that message is long.

v           NetLink sockets can be used for Multicast.

•

  

Standard API's

vThe standard socket APIs—socket(), sendmsg() or sendto(), recvmsg() or recvFrom() and close()—can be used by user-space applications to access netlink socket.

int socket(int domain, int type, int protocol)

 The socket domain (address family) is AF_NETLINK, and the type of socket is either SOCK_RAW or SOCK_DGRAM, because netlink is a datagram-oriented service.

 The protocol (protocol type) selects for which netlink feature the socket is used. The following are some predefined netlink protocol types: NETLINK_ROUTE, NETLINK_FIREWALL, NETLINK_ARPD, NETLINK_ROUTE6 and NETLINK_IP6_FW. You also can add your own netlink protocol type easily.

Standard API's for Kernel Space:

How to Create Netlink socket in kernel?

struct sock* 

netlink_kernel_create(struct net *net,int unit,unsigned int groups, 

                  void (*input)(struct sk_buff *skb), 

                  struct mutex *cb_mutex, 

                  struct module *module)

 e.g:  No need to pass the net parameter.

/* Create NetLink Socket */

nlSock = netlink_kernel_create 
                                 ( NETLINK_MUNISOCK,    // Unit can be Protocol Type
                                    0,                                            // Groups can be Zero
                                   msgFromNetLinkSock,         // Call Back Function
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,23)
                                   NULL,                                      // Mutex can be zero
#endif
                                   THIS_MODULE);
 

 Important Hint: The value of NETLINK_MUNISOCK should be same in User space and Kernel Space 

                                    This is nothing about protocol Type and it should be same in User space and Kernel Space.
    
                                   cat  /proc/net/netlink  -> To check the netlink Sock ID's in kernel

How to send a unicast message to user Space?

 

int
netlink_unicast(struct sock *ssk, struct sk_buff *skb, u32 pid, int nonblock);

e.g: 

int Error;

Error = netlink_unicast(sk, skb, pid, MSG_DONTWAIT);

if(Error == -1)
{
    printk(KERN_INFO "Unable to send Info Back to User Space\n");
}

How to de-queue netlink message from Call back function in Kernel Space?

skb = skb_dequeue(&sk->receive_queue))

E.g: 

static void msgFromNetLinkSock(struct sock *sk, int nLength)

{

             struct sk_buff *skb;
     struct nlmsghdr *nlh = NULL;
     unsigned char *payload = NULL;

     while ((skb = skb_dequeue(&sk->receive_queue)) != NULL) 

     {
         /* process netlink message pointed by skb->data */
         nlh = (struct nlmsghdr *)skb->data;
         payload = NLMSG_DATA(nlh);
         /* process netlink message with header pointed by
          * nlh and payload pointed by payload
          */
      }

}

Working Code For NETLINK Socket: 

Kernel Space Code:

•

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/moduleparam.h>
#include <linux/version.h>
#include <linux/netlink.h>
#include<linux/udp.h>

/* Optional Headers  not Required, If you don't want Remove*/
#include <linux/in.h>
#include <linux/ip.h>
#include <linux/socket.h>
#include <linux/icmp.h>

/* These Headers are also Optional (Not Required),
   But these are Required for NETFILTER functionality */
#include <net/checksum.h>
#include <linux/netfilter.h>
#include <linux/netfilter_ipv4.h>
#include <linux/interrupt.h>

/* NetLink Definitions */
#define NETLINK_HEADER_SIZE     16
#define NETLINK_MUNISOCK        25
#define NL_MSG_TYPE             20
#define MAX_BUFFER_SIZE         1024

/* Driver Definitions */
#define DRIVER_AUTHOR      "AMSEKHAR"
#define DRIVER_DESC        "SAMPLE_NETLINK_SOCKET"

#define MAX_DATA_SIZE      50

/* Dump Function */
static void dumpRecvData(unsigned char *data, unsigned int dataLen);

/* Declare Buffer */
unsigned char *ucBuffer = NULL;

/* Netlink Sock ID*/
static struct sock *nl_sock_id = NULL;

/* Process Id */
static pid_t pid = 0;

/*
 * Function Name : dumpRecvData()
 */
void dumpRecvData(unsigned char *data,  unsigned int len)
{
    unsigned int uIndx=0;

    printk("The Data:\n");
    if(data)
    {
           for(uIndx=0; uIndx<len; ++uIndx)
          {
                 if(uIndx%32 == 0)
                {
                     printk("\n%4d:", uIndx);
                 }
                if(uIndx%4 == 0)
               {
                       printk(" ");
               }
               printk("%02x", data[uIndx]);
         }
    }
    printk(" Length of Bytes: %d\n", len);
    printk("\n");
}

/*
 * Function Name : sendDataToUserSpace()
*/
void sendDataToUserSpace(unsigned char  *ucBuffer, int unDataLen)
{
    /* Intialise the Sk buffer */
    struct sk_buff *skb = NULL;

    /* Intialise the Netlink message Hdr */
    struct nlmsghdr *nlhdr = NULL;

    int nRet= 0;

    /* Allocate the Memory Using SKB for sending To Appl Space */
   skb = alloc_skb((unDataLen + sizeof (struct nlmsghdr)), GFP_ATOMIC);

    /* Move the Tail Pointer at end of the Buffer */
    skb_put(skb, (unDataLen + sizeof (struct nlmsghdr)));

    /* Validate the Skb */
    if(NULL != skb)
    {
        nlhdr = (struct nlmsghdr *)skb->data;

        nlhdr->nlmsg_len = NLMSG_SPACE(MAX_BUFFER_SIZE);
        /*pid=0 corresponds to kernel */
        nlhdr->nlmsg_pid = 0;
        nlhdr->nlmsg_flags = 0;
        nlhdr->nlmsg_seq = 1;
        nlhdr->nlmsg_type = NL_MSG_TYPE;

        /* Copy the Payload */
        memmove(NLMSG_DATA(nlhdr), ucBuffer, unDataLen);
        NETLINK_CB(skb).pid = 0;

        /*BSGATM.exe Process ID */
        NETLINK_CB(skb).dst_pid = pid ;
        NETLINK_CB(skb).dst_group = 0;  /* unicast */

        /* send the NetLink Message to Application space */
      nRet = netlink_unicast(nl_sock_id, skb, pid, MSG_DONTWAIT);

        if(nRet == -1)
        {
            printk("Not Send Successfully\n");
            kfree_skb(skb);
        }
        else
        {
            printk(" Successfully sent to Application space\n");
        }
    }
}

 /*
  * Function Name : MsgFromNetLinkSock
  */
static void msgFromNetLinkSock(struct sock *sk, int len)
{
    struct sk_buff *skb = NULL;
    struct nlmsghdr *nlhdr = NULL;
    int type, unDataLen;
    /* skb_dequeue takes the  buffer from a queue.
     * Nothing(No Buffer) is there then it return  NULL pointer.*/
    while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL)
    {

        /* Extract the NetLink Header fields for Processing */
        nlhdr = (struct nlmsghdr *)skb->data;

        if(nlhdr->nlmsg_pid != 0)
        {
            pid = nlhdr->nlmsg_pid; /*pid of sending process */
        }
        /* Copy the Type of message */
        type = nlhdr->nlmsg_type;

        if (type != NL_MSG_TYPE)
        {
            printk(KERN_INFO "Recieved Message Type :%d", type);
            kfree_skb(skb);
            continue;
        }
        /* Point the data to Buffer */
        ucBuffer = NLMSG_DATA(nlhdr);
        

        /* Compute Actual Length */
        /* skb->len and nlhdr->nlmsg_len are same.*/
        //or unDataLen = nlhdr->nlmsg_len - NETLINK_HDR_SIZE;
        unDataLen = skb->len - NETLINK_HEADER_SIZE;

        /* dump Recieved Data*/
        dumpRecvData(ucBuffer, unDataLen);
        
        /* Send the Data to User SPace from Kernel */
        sendDataToUserSpace(ucBuffer, unDataLen);
   
      /* If this is not there, then it will throw an atomic error */
        kfree_skb(skb);

    }
}

/*
 * Function Name : netlinkProcess_init()
*/

static int __init init_netlinkAppl(void)
{

    printk(KERN_INFO "NFNL Loading netlinkProcess  Module\n");

    // Create NetLink Socket
    nl_sock_id = netlink_kernel_create(NETLINK_MUNISOCK,
                                    0,
                                    msgFromNetLinkSock,
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,22)
                                    NULL,
#endif
                                    THIS_MODULE);
    if(!nl_sock_id)
    {
        printk(KERN_ERR "NFNL %s: receive handler registration failed\n", __func__);
        return -ENOMEM;
    }

    return 0;
}

 /*
 * Function Name : cleanup Module()
 */

static void __exit exit_netlinkAppl(void)
{
    printk(KERN_INFO "NFNL UnLoading netlinkProcess  Module\n");
    if(nl_sock_id)
    {
        //netlink_kernel_release(nl_sock_id);
        sock_release(nl_sock_id->sk_socket);
    }
}

/* Module Init and exit */
module_init(init_netlinkAppl);
module_exit(exit_netlinkAppl);

/* Module properties */
MODULE_LICENSE("Proprietary");
MODULE_AUTHOR(DRIVER_AUTHOR);
MODULE_DESCRIPTION(DRIVER_DESC);
MODULE_VERSION("2.6.18.194");

How to compile the Kernel Space Code

1. Create Makefile

2. Copy the below contents in to Makefile.

obj-m += netlinksock.o
netlinksock-objs := netlinkProcess.o
all:
  make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
  make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean



•

User Space Code:

#include <iostream.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <linux/netlink.h>

/* Netlink Defines */
#define NL_MSG_TYPE         20
#define NETLINK_MUNISOCK    25
#define NETLINK_HEADER_SIZE 16
#define MAX_DATA_SIZE       30

/* Socket FD */
int netlinkSockFD;
struct sockaddr_nl SrcAddr;
struct sockaddr_nl DestAddr;

/* Sequence No */
int SeqNo;
/* process Id */
pid_t PID;

/* Configure The Structure */
unsigned char ucBuffer[MAX_DATA_SIZE];

/*
 * Name            : SendMessage()
 */
bool sendDataToKernel(int len)
{
    unsigned char buf[1024]={0};
    bool ret = false;
    int errno;
    unsigned int uIndx;

    /* Intialise the buffer */
    for(uIndx=0; uIndx<len; uIndx++)
    {
       ucBuffer[uIndx] = 0x31;
    }

    struct nlmsghdr *nlh = (struct nlmsghdr*)buf;

    nlh->nlmsg_len = (len + NETLINK_HEADER_SIZE );
    nlh->nlmsg_pid = PID;  /* self pid */

    nlh->nlmsg_seq = SeqNo++;
    nlh->nlmsg_flags = 0;
    nlh->nlmsg_type = NL_MSG_TYPE;

    /* Copy the Buffer */
    memcpy(NLMSG_DATA(nlh), &ucBuffer, len);

    /* Send the Data */
    int res = sendto(netlinkSockFD, nlh, nlh->nlmsg_len, 0,
                   (const struct sockaddr*) &DestAddr, sizeof(struct sockaddr_nl));

    if(-1 == res)
    {
        cout<<"Unable to Transmit Netlink Message: "<<strerror(errno);
        return ret;
    }
    return (ret=true);
}

/*
 * Function Name : dumpData()
 */
void dumpData(unsigned char *data,  unsigned int len)
{
    unsigned int uIndx;

    if(data)
    {
        for(uIndx=0; uIndx<len; ++uIndx)
        {
           if(uIndx%32 == 0)
           {
              printf("\n%4d:", uIndx);
           }
           if(uIndx%4 == 0)
           {
                 printf(" ");
           }
           //printf("%02x", data[uIndx]);
           printf("%02x", *(data + uIndx));
        }
    }
    printf("\n Length of Bytes: %d\n", len);
    printf("\n");
}
/*
 * Func: recvMsgFromKernel()
 */

void *recvMsgFromKernel(void *args)
{
    unsigned char ucBuffer[1024];
    struct sockaddr_nl client_addr;
    socklen_t size = sizeof(struct sockaddr_nl);
    int length =0;

    while(true)
    {
        memset(ucBuffer, 0,1024);

        // Recieve Data from Kernel
        length = recvfrom(netlinkSockFD, ucBuffer, 1024 , MSG_NOSIGNAL, (struct sockaddr*)&client_addr, &size);
        if (length > 0 )
        {
             struct nlmsghdr *nlh = (struct nlmsghdr*) ucBuffer;
             if(0 != nlh->nlmsg_pid)
             {
                 printf("\nReceived Message from Unknown Source\n");
                 continue;
             }
             else
             {
                 unsigned char ucBuf[1024];
                 unsigned int unActualLen = length - sizeof (struct nlmsghdr);
                 memmove(&ucBuf, NLMSG_DATA(nlh), length - sizeof(struct nlmsghdr));
                 /* dump Recieved Data */
                dumpData(ucBuf, unActualLen);
             }
         }
    }
}

 /*
 * Name            : main()
 */
int main()
{
    bool ret;
    int ch, len, errno;

    /* Intialise Socket Paramters */
    netlinkSockFD = -1;
    PID = getpid();
    SeqNo = 0;
    memset(&DestAddr, 0, sizeof(struct sockaddr_nl));
    memset(&SrcAddr, 0, sizeof(struct sockaddr_nl));
          
          //create receive thread
    pthread_t threadId;

    if(pthread_create(&threadId, NULL, recvMsgFromKernel, NULL))
    {
        perror("Error in creating receiver thread");
        return -1;
    }

    /* Create NetLink Socket */
    netlinkSockFD = socket(PF_NETLINK, SOCK_RAW, NETLINK_MUNISOCK);
    if(netlinkSockFD == -1)
    {
        cout<< "Unable to open socket - " << strerror(errno);
        return 0;
    }
   else
    {
       cout<< "Netlink Socket created Successfully\n";
    }

    memset(&DestAddr, 0, sizeof(struct sockaddr_nl));
    DestAddr.nl_family = AF_NETLINK;
    DestAddr.nl_pid = 0; /* For Linux Kernel */
    DestAddr.nl_groups = 0; /* unicast */

    do
    {
       cout<<endl;
       cout<<" Enter your choice:\t"<<endl;
       cout<<" 1. Send Data to Kernel" <<endl;
       cout<<" 2. exit" <<endl;
       cin>>ch;
       cout<<endl;

       switch(ch)
       {
           case 1:
              {
                   cout<<"Enter the Length of the Payload "<<endl;
                   cin>>len;

                   ret = sendDataToKernel(len);
                   if(ret)
                   {
                        cout<<"\n Netlink Socket Sent successfully \n";
                   }
                   else
                   {
                        cout<<"\n Netlink Socket Failed in Sending Message \n";
                   }
                   break;
               }
           default:
                  cout<<"Invalid Choice\n";
                  break;
       }
    }while(ch!=2);
}

How to compile the user space code:

> First save this file as userNetLinkSock.cpp then compile using below command

> c++ userNetLinkSock.cpp -lpthread -o user

 

 TIPS For NetLink Socket:

> While allocating memory for SKB,   once allocate memory , after that "skb_put" to be there to move the tail pointer appropriately.

    e.g:
    skb = alloc_skb((unDataLen + sizeof (struct nlmsghdr)) , GFP_ATOMIC);

    /* Move the Tail Pointer at end of the Buffer */
    skb_put(skb, (unDataLen + sizeof (struct nlmsghdr)));

> Set the destination PID value(while sending data to user space) appropriately depending on the receiving application.

> To avoid this kind of error :assertion (!atomic_read(&sk->sk_rmem_alloc)) failed, ensure that the skb is freed after being dequeued and processed.  ( SKB Should be freed in De-queue Function)       e.g: static void msgFromNetLinkSock(struct sock *sk, int len)                                                                  {
     /* skb_dequeue takes the  buffer from a queue.
      * Nothing(No Buffer) is there then it return  NULL pointer.*/
    while ((skb = skb_dequeue(&sk->sk_receive_queue)) != NULL)
    {

        /* Send the Data to User SPace from Kernel */
        xxfunction(ucBuffer, unDataLen);

        kfree_skb(skb);   // free the SKB
    }}                          

> While sending the data from user space, the net link message length should be                           actual data len+  NETLINK Header size 

e.g:  nlh->nlmsg_len = (len+NETLINK_HEADER_SIZE(16)

> The payload of any buffer should be stored in NLMSG_DATA(nlhdr).

e.g: while receiving data from user space , then point the   ucBuffer = NLMSG_DATA(nlhdr);           

e.g:   While sending data from kernel,                                                                                                                /* Copy the Payload */
        memmove(NLMSG_DATA(nlhdr), ucBuffer, unDataLen);
                                                                       

For Further reading:   http://people.ee.ethz.ch/~arkeller/linux/kernel_user_space_howto.html