melfneerg.com
- 'cos life is like that
[Blog]
[Pictures]
[Links]
[About]
About the Author
Tudor Davies
Tudor is a techie turned manager who fights like mad to keep his tech skills honed and relevant. Everything from web hosting, networking, *nix and the like. Constantly developing and co-ordinating with others to make the web a better (and easier to use) place.
Tuesday, 15th Jan 2013
Posted @ 10:31
Following a discussion on a couple of mailing lists about backpacks (started because I needed a new one as the zip had gone into FUBAR mode and my laptop had actually fallen out at one point), I thought I would share what I carry with around with me in my backpack:
Dell D630 (14.1", T7250 2.0Ghz, 2Gb RAM, 160GB) in a sleeve
Dell PSU
Varying size USB sticks
Ethernet Xover cable
Ethernet cable
Pen & Pencil
Notepad
HP Touchpad in case
The HP Touchpad case is pretty cool don't you think? Its designed for an iPad but it fits perfectly.
The case is a Cocoon CPG36 and holds:
I also need to carry around the following with me, so I am going to buy another Grid-IT! but a bit smaller:I'm likely to get the CPG10 as I think that will hold everything I need it to.
[ 1 comments : Add | Read ]
Monday, 14th Jan 2013
Posted @ 17:15
So a RAQ3 performs pretty poorly even with Lighttpd and PHP (FastCGI). I put one of my sites online which is pretty mySQL heavy and essentially I could bring the machine to its knees with about 3 users loading the index page :)
I use openLoad if I need to test from a Windows box, so I fired it up and aimed it at the RAQ3. The results were not pretty:
Machine booted up and caches cleared - 38mb in use, CPU usage at 3.8% - nice.
openLoad pretending to be 5 clients hitting index.php (which makes 3 mySQL lookups) for 30 seconds:
CPU hit 100% and stayed there. openLoad reported back:-
Total TPS: 1.13
Avg. Response Time: 3.139 sec.
Max Response Time: 11.935 sec
Total Requests: 34
Total Errors: 0
The answer? Varnish. Now I've used Varnish before for doing some HA/Load Balancing testing and that was fine. But I haven't played around too much with its reverse proxying capabilties. I also wasn't expecting much of an improvement, this is an AMD K6-2 300 we are talking about.
So I installed it using the instructions found here, changed the config so that it was listening on port 80, changed Lighttpd to listen on port 8080 and started/restarted services.
Machine booted up and caches cleared - 38mb in use, CPU usage at 3.8% - all the same as before.
openLoad pretending to be 5 clients hitting index.php (which makes 3 mySQL lookups) for 30 seconds:
CPU hits 100% and then hovers around 20%. openLoad reported back:-
Total TPS: 21.85
Avg. Response Time: 0.225 sec.
Max Response Time: 1.929 sec
Total Requests: 656
Total Errors: 0
Erm - thats a bit better! Quite an improvement...
[ no comments : Add ]
Monday, 14th Jan 2013
Posted @ 15:06
I have had a couple sat unused for some time and I thought it would be worthwhile turning them into something fairly useable. So here is what I did:
i/ Installed Strongbolt 1.06-03 (booted it inside VirtualBox with a bridged ethernet adapter - worked a treat)
ii/ Followed the instructions here upto and including Step 4 to get the box to CentOS 5.8 (Final)
iii/ Followed the instructions here to get Lighttpd, PHP and mySQL installed.
iv/ Followed the instructions here to upgrade PHP to 5.3.3 (believe me, this is worthwhile)
v/ Tweaked the hell out of startup. The output of
chkconfig --list
produces this:
auditd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
dc_client 0:off 1:off 2:off 3:off 4:off 5:off 6:off
dc_server 0:off 1:off 2:off 3:off 4:off 5:off 6:off
gpm 0:off 1:off 2:off 3:off 4:off 5:off 6:off
haldaemon 0:off 1:off 2:off 3:off 4:off 5:off 6:off
httpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
iscsi 0:off 1:off 2:off 3:off 4:off 5:off 6:off
iscsid 0:off 1:off 2:off 3:off 4:off 5:off 6:off
kdump 0:off 1:off 2:off 3:off 4:off 5:off 6:off
kudzu 0:off 1:off 2:off 3:off 4:off 5:off 6:off
lcdsleep.init 0:off 1:off 2:off 3:on 4:off 5:off 6:off
lighttpd 0:off 1:off 2:on 3:on 4:off 5:on 6:off
lvm2-monitor 0:off 1:on 2:off 3:off 4:off 5:off 6:off
mcstrans 0:off 1:off 2:off 3:off 4:off 5:off 6:off
mdmonitor 0:off 1:off 2:off 3:off 4:off 5:off 6:off
mdmpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
messagebus 0:off 1:off 2:off 3:off 4:off 5:off 6:off
multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
mysqld 0:off 1:off 2:on 3:on 4:off 5:on 6:off
named 0:off 1:off 2:off 3:off 4:off 5:off 6:off
netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off
netfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off
netplugd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nscd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
ntpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
portmap 0:off 1:off 2:off 3:on 4:on 5:on 6:off
psacct 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rawdevices 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rdisc 0:off 1:off 2:off 3:off 4:off 5:off 6:off
restorecond 0:off 1:off 2:off 3:off 4:off 5:off 6:off
saslauthd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
sendmail 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
syslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
and the output of
free -m && ps aux
after a reboot produces this:
total used free shared buffers cached
Mem: 503 63 440 0 3 37
-/+ buffers/cache: 22 481
Swap: 976 0 976
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 1.0 0.1 2172 664 ? S 14:45 0:00 init [3]
root 2 0.0 0.0 0 0 ? SN 14:45 0:00 [ksoftirqd/0]
root 3 0.0 0.0 0 0 ? S< 14:45 0:00 [events/0]
root 4 0.0 0.0 0 0 ? S< 14:45 0:00 [khelper]
root 5 0.0 0.0 0 0 ? S< 14:45 0:00 [kthread]
root 7 0.0 0.0 0 0 ? S< 14:45 0:00 [kblockd/0]
root 10 0.0 0.0 0 0 ? S< 14:45 0:00 [khubd]
root 66 0.0 0.0 0 0 ? S 14:45 0:00 [pdflush]
root 67 0.0 0.0 0 0 ? S 14:45 0:00 [pdflush]
root 69 0.0 0.0 0 0 ? S< 14:45 0:00 [aio/0]
root 68 0.0 0.0 0 0 ? S 14:45 0:00 [kswapd0]
root 142 0.0 0.0 0 0 ? S< 14:45 0:00 [kseriod]
root 196 0.0 0.0 0 0 ? S 14:45 0:00 [kjournald]
root 250 0.6 0.1 2396 620 ? S s 14:45 0:00 /sbin/udevd -d
root 570 0.1 0.0 0 0 ? S 14:45 0:00 [kjournald]
root 572 0.0 0.0 0 0 ? S 14:45 0:00 [kjournald]
root 574 0.0 0.0 0 0 ? S 14:45 0:00 [kjournald]
root 800 0.0 0.0 2424 484 ? Ss 14:45 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.p
root 833 0.1 0.1 1828 588 ? Ss 14:45 0:00 syslogd -m 0
root 836 0.0 0.0 1776 400 ? Ss 14:45 0:00 klogd -x
rpc 845 0.0 0.1 1928 544 ? Ss 14:45 0:00 portmap
root 865 0.0 0.2 7276 1056 ? Ss 14:45 0:00 /usr/sbin/sshd
root 881 0.6 0.6 10104 3136 ? Ss 14:45 0:00 sshd: root@pts/0
root 910 0.3 0.2 2564 1172 ? S 14:45 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --l
mysql 960 0.3 0.7 22164 3784 ? Sl 14:45 0:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run
root 988 0.0 0.3 9476 1680 ? Ss 14:45 0:00 sendmail: accepting connections
smmsp 998 0.0 0.2 8316 1496 ? Ss 14:45 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 1015 0.4 0.2 2568 1360 pts/0 Ss 14:45 0:00 -bash
lighttpd 1041 0.0 0.1 5368 852 ? S 14:45 0:00 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
lighttpd 1047 0.7 1.2 56536 6704 ? Ss 14:45 0:00 /usr/bin/php-cgi
lighttpd 1048 0.8 1.2 56532 6696 ? Ss 14:45 0:00 /usr/bin/php-cgi
lighttpd 1051 0.7 1.2 56532 6700 ? Ss 14:45 0:00 /usr/bin/php-cgi
lighttpd 1052 0.8 1.2 56532 6700 ? Ss 14:45 0:00 /usr/bin/php-cgi
root 1053 0.0 0.2 3352 1108 ? Ss 14:45 0:00 crond
root 1060 0.0 0.0 1752 384 ? S 14:45 0:00 /sbin/lcdsleep
root 1067 0.0 0.1 1772 516 ttyS0 Ss+ 14:45 0:00 /sbin/agetty ttyS0 115200 vt100-nav
lighttpd 1073 0.0 0.4 56532 2460 ? S 14:45 0:00 /usr/bin/php-cgi
lighttpd 1074 0.0 0.4 56536 2464 ? S 14:45 0:00 /usr/bin/php-cgi
lighttpd 1075 0.0 0.4 56532 2460 ? S 14:45 0:00 /usr/bin/php-cgi
lighttpd 1076 0.0 0.4 56532 2456 ? S 14:45 0:00 /usr/bin/php-cgi
root 1102 0.0 0.1 2288 844 pts/0 R+ 14:46 0:00 ps aux
vi/ Followed the instructions here to get PEAR installed and then issued
yum install pcre-devel
pecl install apc
service lighttpd restart
to get apc (Advanced PHP Cache) installed and running.
So far everything is running well. I wouldn't say it was fast but perfectly acceptable for a dev environment. Next up, compile and install Varnish and push the throughput up :)
[ no comments : Add ]
Thursday, 10th Jan 2013
Posted @ 15:25
Wrote this little beauty this morning:
SELECT
CONCAT('0', called_number) AS called,
calling_number AS calling,
CONCAT(DATE_FORMAT(startdate,'%d/%m/%Y'),' ',starttime) AS start,
CONCAT(DATE_FORMAT(startdate,'%d/%m/%Y'),' ',ADDTIME(starttime,sec_to_time(3))) AS connect,
CONCAT(DATE_FORMAT(startdate,'%d/%m/%Y'),' ',ADDTIME(starttime,sec_to_time(duration+3))) AS end,
duration
FROM cdr
WHERE duration != '0'
AND calling_number NOT IN (
SELECT sipnum FROM sip_numbers)
AND CONCAT('0',called_number) NOT IN (
SELECT sipnum FROM sip_numbers);
[ no comments : Add ]
Friday, 21st Dec 2012
Posted @ 10:51
This morning, I rebooted my HP Touchpad into WebOS mode and removed everything I could from it.
I haven't used it in WebOS mode for many, many months, so I decided it was time to remove all the apps, games and data I had installed, installed any updates that were available and rebooted it back in Android. This left it with just some development apps and the stuff that came with it but freed up a few Gbs of space for me to fill up with Android apps and games :)
[ no comments : Add ]
Thursday, 20th Dec 2012
Posted @ 15:19
I have been playing around with Tiny Core and Micro Core Linux for quite a while now but have yet to do anything particularly useful with it.
Well now I have :)
I have created a minimal VMWare image capable of being a complete (and useable) webserver - a LEMP server - its only a 60mb download and the links are further down this post.
Tech specs:
1 vcpu
256mb RAM
1Gb hard drive (split 256mb swap, the rest as storage)
Software installed:
Core 4.7.1
openSSH 6.0p1
nGinx 1.0.12
PHP 5.3.8 (fast_cgi)
mySQL 5.5.17
along with all dependencies.
There is persistence for all the config files for the above and the mySQL databases. to be sure, issue sudo filetool.sh -b if you make any changes to anything.
- Access is limited to SSH/SCP
- all web files must be stored under
/usr/local/html/ - The network is set to DHCP
- System
Login: tc
Password: youshouldchangethis - mySQL
Login: root
Password: youshouldchangethis
It has been tested under ESXi 4.1 and also VM Player with no problems (apart from maybe complaints about floppy drives!)
Once its up and running and seen a few iterations from ApacheBench, htop shows this:
I am regularly seeing it capable of processing over 1000 requests a second (hitting a page that calls the phpinfo() function) running on a single 2.3Ghz vcpu
[root@365 ~]# ab -n 2000 -c 500 http://172.18.11.20/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 172.18.11.20 (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests
Server Software: nginx/1.0.12
Server Hostname: 172.18.11.20
Server Port: 80
Document Path: /index.php
Document Length: 169 bytes
Concurrency Level: 500
Time taken for tests: 1.769 seconds
Complete requests: 2000
Failed requests: 0
Write errors: 0
Non-2xx responses: 2010
Total transferred: 641190 bytes
HTML transferred: 339690 bytes
Requests per second: 1130.60 [#/sec] (mean)
Time per request: 442.244 [ms] (mean)
Time per request: 0.884 [ms] (mean, across all concurrent requests)
Transfer rate: 353.97 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 2 91 243.2 24 1011
Processing: 2 145 389.9 15 1662
Waiting: 1 142 388.6 14 1660
Total: 5 236 454.3 47 1755
Percentage of the requests served within a certain time (ms)
50% 47
66% 59
75% 112
80% 144
90% 1018
95% 1339
98% 1751
99% 1753
100% 1755 (longest request)
Next up, I will be attempting to create a minimal load balancer/reverse proxy (using the likes of Pen and Varnish) appliance and also a Lightttpd version of this appliance.
Anyway, you can find the zip file here or here. if you find it useful or have any questions, leave a comment.
[ no comments : Add ]
Tuesday, 4th Dec 2012
Posted @ 17:07
Had a disk failure recently on the NAS box and lost a chunk of backups - fortunately, all the other boxes remained up, so not all was lost.
Also moved to a new IP range this week (as my employer now has transit via their own POP in Telehouse).
Installed a couple of things on my self-hosting setup:
Suhosin
APC
That should make things a little more secure and also a little faster (even just using the byte-code cache)
If all goes well, I shall be rebuilding one of the web servers here in the office (its already on an ESXi setup, so firing up a new one should not be too bad!) with my new found knowledge and securing/speeding that up too :)
I am also planning on moving my home server from the dining room (bottom of shelves behind things) into the sun lounge and from a P4 2.0Ghz Dell Optiplex SX260 (w/ 512mb RAM) onto a rack mounted Celeron 2.4Ghz (w/ 1.5Gb RAM). Should be a matter of swapping the hard disks over. Try that with a windows box!
[ no comments : Add ]
Wednesday, 22nd Aug 2012
Posted @ 09:30
I now have a case for my Raspberry Pi - its one that mounts it onto a VESA plate for attaching to the back of my TV.
Once a stable release of Xbian or Raspbmc is released, I will update to it and mount it on the bedroom TV - this means we will be able to watch HD content in the bedroom at long last!
My 2 WR703Ns are up and running - one as my own "FreeFile" project (much like PirateBox) and the other is currently running PirateBox. Although this is due to be changed to run as an access-point so I can have Wifi for my iPhone and Android tablet wherever I can find an ethernet connection :)
Oh yeah - Android Tablet. I took my beloved HP Touchpad running WebOS 3.0.5 and installed CynogenMod 9 - Android ICS 4.0.4
Crikey its slick! The microphone and the camera do not work but when they do - I will likely wipe all my data off the WebOS partition and move to using Android exclusively. Mind you, being able to swap over on a reboot is pretty nice.
mySQL indexes - on this here blog I was getting reports from the SQL server of a slow query. The query was
SELECT COUNT(*) AS total FROM comments WHERE article = "xxx"
and this was being reported as taking over a second! The reason - no index. They was a primary key but no index. A quick drop into the command line and a
CREATE INDEX id_comment ON comment(id,article)
sorted it out. Watch your indices people :)
[ no comments : Add ]
Thursday, 28th Jun 2012
Posted @ 14:21
Long time no see - thats a line that all my blogs seem to say eventually. Life is just too damn busy these days...
1 - I went onto the IRC channel I used to frequent daily over many, many years today and no-one was there. It made me a little sad :(
2 - My littlest one is now 8 months old. Time has flown by! He has a tooth, does a zombie crawl and has just mastered the art of pulling himself up on furniture.
3 - Managed to get a Raspberry Pi but have yet to do anything majorly constuctive with it as I have been playing around with other stuff. XMBC on it is pretty cool (although slow in places) and will only improve as it gets tweaked going forward.
4 - I taught myself PHP:PDO in order to convert some code that used flat files into SQLite
5 - Got hold of a TP-LINK WR703N for £15.29 on eBay (I actually have 2 now). After seeing the results from piratebox I decided to make my own from scratch based on the same ideas but with all my own code. I will release all the code, instructions and images here when it is finalised. Taught myself quite a bit in the process. The second one is going to become a miniature file server for home to replace a PC I currently use once I am sure it can keep up with streaming MKVs around a wired network. iPerf performance puts it at around 90+mbits/s so it should handle it fairly well.
[ no comments : Add ]
Wednesday, 29th Feb 2012
Posted @ 09:51
Been working on speeding up some of my websites recently.
Step 1 - minify all your CSS and JS files
Step 2 - add compression to the .htaccess file.
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
Step 3 - sort out slow mysql queries and/or add indices to tables
Step 4 - dont use RAND() in mySQL queries
There are 2 code snippets below that I ran using "time php -q x.php" and the results are posted below:
1.php
$hostName = "localhost";
$userName = "blah";
$password = "blahblah";
$dbName = "blah";
mysql_connect($hostName, $userName, $password) or die("Unable to connect to host");
mysql_select_db($dbName) or die("Unable to select database $dbName");
$query = "SELECT * FROM Catalog ORDER BY RAND() LIMIT 0, 100";
$result = mysql_query($query) or die(mysql_error());
mysql_close();
2.php
$hostName = "localhost";
$userName = "blah";
$password = "blahblah";
$dbName = "blah";
mysql_connect($hostName, $userName, $password) or die("Unable to connect to host");
mysql_select_db($dbName) or die("Unable to select database $dbName");
$result = mysql_query("SELECT COUNT(*) FROM Catalog");
while ($row = mysql_fetch_array($result)) {
$total_recs = $row["COUNT(*)"];
}
$exnums = """;
for ($cnt=0; $cnt<=100; $cnt++) {
$exnums .= rand(0, $total_recs)."","";
}
$exnums = substr($exnums, 0, -2);
$query = "SELECT * FROM Catalog WHERE ID IN (".$exnums.");";
$result = mysql_query($query) or die(mysql_error());
mysql_close();
The results?
time php -q 1.php
real 0m2.561s
user 0m1.157s
sys 0m0.058s
time php -q 2.php
real 0m1.234s
user 0m1.166s
sys 0m0.057s
mySQL RAND() is officially rubbish!
My site now returns results fast enough to keep up with a steady click on the button :)
[ no comments : Add ]
layout and initial css based on the
Qtractor page