Don't Worry, Be Happy

Apache httpd combined with grpc failed

Posted on | In |

background: we have several apache httpd c++ servers running internally. UI will send request to one of the servers, then the main server will query many other internal servers to aggregate the result. It is one use case of rpc framework, which is more light weight than httpd. Our code is there, I just want to migrate our server to grpc.

Basic Ideas

Keep the main httpd server, which need to accept interal request from application/UI team. Once the httpd server get a message, it will use grpc to send to several protobug message to rpc servers, wait all the result, aggregate and finally return the response.

So I tried to create a client in httpd mod. Just add grpc things in our original code.

Install grpc and protobuf 3

install c++ version grpc. Protobuf 3 is needed. According to the instructions, when installing, it will detect if you already installed protobuf 3. If not, it will install protobuf in side third_party folder. We can also install some latest version of protobuf 3 by instructions

All kinds of errors

Test code is done. I ran into a lot of tricky problems.

compile error

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
In file included from CABRCU.h:1:0,
                 from CABEvaluatorModule.cpp:1:
/usr/include/httpd/httpd.h:451:12: error: expected identifier before numeric constant
 #define OK 0                    /**< Module has handled this stage. */
            ^
/usr/include/httpd/httpd.h:451:12: error: expected ‘}’ before numeric constant
/usr/include/httpd/httpd.h:451:12: error: expected unqualified-id before numeric constant
In file included from /usr/local/include/grpc++/impl/codegen/status.h:38:0,
                 from /usr/local/include/grpc++/impl/codegen/core_codegen_interface.h:38,
                 from /usr/local/include/grpc++/impl/codegen/client_context.h:57,
                 from /usr/local/include/grpc++/impl/codegen/call.h:43,
                 from /usr/local/include/grpc++/impl/call.h:37,
                 from /usr/local/include/grpc++/channel.h:39,
                 from /usr/local/include/grpc++/grpc++.h:56,
                 from /home/changyungong/git/neptune/nise/rpc/audience/CABQueryClient.h:5,
                 from CABEvaluatorModule.cpp:23:
/usr/local/include/grpc++/impl/codegen/status_code_enum.h:150:1: error: expected declaration before ‘}’ token
 }  // namespace grpc
 ^

Looks like the header files of apache httpd and grpc have a conflict.
That is because httpd has the code:

1
#define OK 0                    /**< Module has handled this stage. */

Meanwhile, grpc has the enum:

1
2
3
4
5
6
namespace grpc {
enum StatusCode {
    OK = 0,
    ...
}
}

So that the compiler replace OK in grpc to 0, which equals 0 = 0 for grpc code:

1
2
3
4
5
6
7
8
xx.cc:4:5: error: expected identifier before numeric constant
0 = 0
^
xx.cc:4:5: error: expected ‘}’ before numeric constant
xx.cc:4:5: error: expected unqualified-id before numeric constant
xx.cc:6:1: error: expected declaration before ‘}’ token
}
^

solution:

1
2
// #define OK 0
const int OK = 0;

Have to change httpd header file and recompile.

protobuf version error

protobuf higher version is not compatipable with low version, since the encode method may be changed.
error:

1
2
3
[libprotobuf FATAL google/protobuf/stubs/common.cc:78] This program was compiled against version 3.0.0 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.2.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "src/cpp/ext/reflection.pb.cc".)
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  This program was compiled against version 3.0.0 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.2.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "src/cpp/ext/reflection.pb.cc".)

I installed grpc first, then install a newer version of protobuf v3.2.0. Noticed that in grpc repo, under third party folder, the version of protobug is 3.0.0. So for the safely use the solution is to reinstall protobuf v3.0.0 s(The same version used under somepath/grpc/third_party/protobuf ).

Finally I ran into an unsolved problem.

1
2
3
4
[libprotobuf ERROR google/protobuf/descriptor_database.cc:57] File already exists in database: ua_result_rpc.proto
[libprotobuf FATAL google/protobuf/descriptor.cc:1164] CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: generated_database_->Add(encoded_file_descriptor, size):

The reason may be that according to the design of grpc, protobuf .so library cannot br loaded more than once. Some people said to link rpc and protobuf static library.
But in my project, due the makefile and the code infrustructure, cannot make it out.

This may be why in ORC Client c++, which also uses protobuf, protobuf is lined in .a.