This is the first time I am posting a question here so please ignore deviations from standard format...
Here is my setup. Nginx --> Uwsgi --> Django --> (Memcached/ Mysql)
I have a set of APIs in Django which return json response. Earlier I was using memcached 2 part middleware to do auto caching. Now to get more control ,I have moved to low level APIs. I use celery to create pickled data sets and insert data in memcached once a day and My Apis fetch the data from memcached.
I am using NGINX and Uwsgi( 16 workers) in my production environment and was getting a speed of about 8K requests with 2 part middleware approach. Only my first request used to be slow. But with lower level cache I am getting 1/10th speed .
Initially I thought it was due to extra middlewares and custom code which was getting executed, but I have trimmed that also. I moved from pickle to cPickle but I do not think that is the bottleneck. I am using apache benchmark to conduct tests.
I am attaching Memcached ( -vv) logs for both middleware approach and for low level cache api. If anybody can point me right direction........ thanks
I saw catche-middleware code for Django. They seem to be saving django httpresponse object in cache. Middleware also calls low-level apis internally .I tried saving httpresponse in cache also , but no change...
LOGS
<pre>
MIDDLEWARE approach memcached LOGS
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>34 END
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>35 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>36 END
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>40 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<34 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>34 END
<38 new auto-negotiating client connection
<42 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>38 END
42: Client using the ascii protocol
<42 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>42 END
>37 END
<41 new auto-negotiating client connection
<33 connection closed.
<36 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>36 END
41: Client using the ascii protocol
<41 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>41 END
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>39 END
<38 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>38 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<31 connection closed.
<36 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<39 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>39 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<35 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>35 END
<33 connection closed.
<30 connection closed.
<42 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>42 END
<37 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>37 END
<35 connection closed.
<39 connection closed.
<32 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
<30 new auto-negotiating client connection
>31 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>31 END
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<32 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>32 END
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<31 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>31 END
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>33 END
<33 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>33 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>30 END
<38 connection closed.
<32 connection closed.
<30 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>30 END
<33 connection closed.
<37 connection closed.
<31 connection closed.
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>29 END
<34 connection closed.
<42 connection closed.
<41 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>41 END
<40 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>40 END
<28 connection closed.
<30 connection closed.
<29 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>29 END
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>28 END
<28 get :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 sending key :1:views.decorators.cache.cache_page..GET.6f43e949c0c0683561d169117dac1088.d41d8cd98f00b204e9800998ecf8427e.en-us.Asia/Kolkata
>28 END
<30 new auto-negotiating client connection
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 sending key :1:views.decorators.cache.cache_header..6f43e949c0c0683561d169117dac1088.en-us.Asia/Kolkata
>32 END
<40 connection closed.
</pre>
----------
----------
LOW level logs
<pre>
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||
>32 END
<36 connection closed.
<36 new auto-negotiating client connection
36: Client using the ascii protocol
<36 get :1:ls|1|0|10|||
>36 sending key :1:ls|1|0|10|||
>36 END
<40 connection closed.
<40 new auto-negotiating client connection
40: Client using the ascii protocol
<40 get :1:ls|1|0|10|||
>40 sending key :1:ls|1|0|10|||
>40 END
<34 connection closed.
<34 new auto-negotiating client connection
34: Client using the ascii protocol
<34 get :1:ls|1|0|10|||
>34 sending key :1:ls|1|0|10|||
>34 END
<29 connection closed.
<29 new auto-negotiating client connection
29: Client using the ascii protocol
<29 get :1:ls|1|0|10|||
>29 sending key :1:ls|1|0|10|||
>29 END
<35 connection closed.
<35 new auto-negotiating client connection
35: Client using the ascii protocol
<35 get :1:ls|1|0|10|||
>35 sending key :1:ls|1|0|10|||
>35 END
<37 connection closed.
<37 new auto-negotiating client connection
37: Client using the ascii protocol
<37 get :1:ls|1|0|10|||
>37 sending key :1:ls|1|0|10|||
>37 END
<41 connection closed.
<41 new auto-negotiating client connection
41: Client using the ascii protocol
<41 get :1:ls|1|0|10|||
>41 sending key :1:ls|1|0|10|||
>41 END
<39 connection closed.
<39 new auto-negotiating client connection
39: Client using the ascii protocol
<39 get :1:ls|1|0|10|||
>39 sending key :1:ls|1|0|10|||
>39 END
<38 connection closed.
<38 new auto-negotiating client connection
38: Client using the ascii protocol
<38 get :1:ls|1|0|10|||
>38 sending key :1:ls|1|0|10|||
>38 END
<30 connection closed.
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:ls|1|0|10|||
>30 sending key :1:ls|1|0|10|||
>30 END
<31 connection closed.
<31 new auto-negotiating client connection
31: Client using the ascii protocol
<31 get :1:ls|1|0|10|||
>31 sending key :1:ls|1|0|10|||
>31 END
<42 connection closed.
<42 new auto-negotiating client connection
42: Client using the ascii protocol
<42 get :1:ls|1|0|10|||
>42 sending key :1:ls|1|0|10|||
>42 END
<28 connection closed.
<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 get :1:ls|1|0|10|||
>28 sending key :1:ls|1|0|10|||
>28 END
<43 connection closed.
<43 new auto-negotiating client connection
43: Client using the ascii protocol
<43 get :1:ls|1|0|10|||
>43 sending key :1:ls|1|0|10|||
>43 END
<33 connection closed.
<33 new auto-negotiating client connection
33: Client using the ascii protocol
<33 get :1:ls|1|0|10|||
>33 sending key :1:ls|1|0|10|||
>33 END
<32 connection closed.
<34 connection closed.
<32 new auto-negotiating client connection
32: Client using the ascii protocol
<32 get :1:ls|1|0|10|||
>32 sending key :1:ls|1|0|10|||
</pre>
