Pencarian

Rss Posts

 

 

 

Berita pada kategori ‘GNU/Linux’

MepSQL Debs for Ubuntu now released – courtesy of cool tweaks to the build system.

Feb 19, 2011

After another week of hacking on MepSQL the DEB files for Ubuntu are now available.(MepSQL is my new “just a hobby” MySQL fork project.)
The Download page has instructions on how to install the packages with a simple apt-get install command. Debian packages will appear soon as they are now easy to add – I mostly just need to add new Amazon images for each.
read more

Roll Your Own Cloud (from linux.conf.au 2011)

Feb 04, 2011

At linux.conf.au 2011 in Brisbane, I had the honor of co-presenting a talk on Rolling Your Own Cloud with SUSE’s Tim Serong. Take a look!

Ubuntu Upstart for automatic MySQL start and stop

Dec 25, 2010

Here at Recorded Future we use Ubuntu (running on Amazon EC2), but so far we have not explored Ubuntu Upstart that much. During the holidays I made an effort to get acquainted with Upstart and to implement proper MySQL start and stop with it.If you do not know Upstart, this is the way you start and stop services in Ubuntu, and it serves the same purpose as the old /etc/init.d scripts, but are a bit more structured and powerful. That said, Upstart is regrettably far from complete, although the functionality is much better and Upstart has some cool features, some things do not work that well. For one thing, documentation, where it exists, is useless, at best. Secondly, there is very limited ability to test and develop Upstart scripts. And this is made worse by the fact that the documentation is so bad. Another thing is that Upstart insist on stopping services, by default, by sending a brutal kill signal. Not good for databases, mostly.In the /etc/init directory are the Upstart scripts you have. In difference to the old init.d scripts, you cannot disable a service in Upstart curenntly. If it is in /etc/init it will be started at system start. That’s it. And this is something that I am sure will be fixed, but for now, again, is something we have to live with. Upstart scripts have the suffix .conf (don’t ask me why), so the default MySQL Upstart script, for example, is called /etc/init/mysql.conf.In an Upstart script, there are Stanzas that determine what to do. Like the exec Stanza that runs a program for example. And you may then ask, when is it run? Startup? Shutdown? And the answer is startup. For shutting things down, as I said before, Upstart will by default just send a kill -9 signal.The minimal startup script you can have, and this actually works in a reasonable way, is to just have one line with an exec stanza, like this:exec /usr/bin/mydaemonWhich will start the daemon. For stopping the daemon, Upstart will send a -9 signal to the started process by default, and nothing more is needed in the Upstart script.For MySQL, we need to make things a bit more complicated. The default mysql.conf Upstart script really is not good. For one thing, it will not do a controlled shutdown of MySQL (this is possible even if Upstart will eventually send a kill -9 anyway). Secondly, this script assumes that what we use is a standard Ubunty installed MySQL distribution, so if you have installed MySQL in /usr/bin/mysql5147 or somethings like that, you are out of luck.So what I wanted to create was an Upstart script for MySQL that fullfilled these requirements:Starts MySQL automatically.Waits for MySQL to be available before exiting.Be configurable to support different MySQL install locations, data directories etc.Do a clean shutdown of MySQL when stopping the MySQL services.Before I show you what I ended up with, I want to comment on the points 2 and 4 above. With Upstart, you can define a script or command to run just before or after a services has been started or stopped, and this is what I use to wait for MySQL to become available, and to send a SIGTERM to the MySQL Server when stopping (which will do a clean MySQL shutdown).So here we go, a complete MySQL Upstart script, the way I want it to work:## MySQL Service for Recorded Future#description “MySQL Server”author “Anders Karlsson, Recorded Future”start on (net-device-up and local-filesystems and runlevel [2345])stop on runlevel [016]expect forkkill timeout 2# Set variables.env MYSQL_ETC=/etc/mysqlenv MYSQL_PIDFILE=/var/run/mysql.pidenv MYSQL_HOME=/usr/local/mysql5.5env MYSQL_INSTANCE=myumask 007exec $MYSQL_HOME/bin/mysqld_safe –defaults-file=$MYSQL_ETC/$MYSQL_INSTANCE.cnf >> /tmp/x.out &post-start script loop=600# Wait for MySQL to start. while [ $loop -gt 0 ]; do if $MYSQL_HOME/bin/mysqladmin –defaults-file=$MYSQL_ETC/$MYSQL_INSTANCE.cnf ping; then break fi loop=$(($loop – 1)) sleep 1 done exit 0end script# Send a soft SIGTERM to MySQL before Upstart will kill it.# A Sigterm to mysqld will cause a controlled shutdown.pre-stop script exec kill -SIGTERM `cat $MYSQL_PIDFILE`# Wait for MySQL to end. Flushing buffers and all. loop=600 while [ $loop -gt 0 ]; do# If the pidfile is found, then continue waiting. if [ -e $MYSQL_PIDFILE ] ; then loop=$((loop – 1)) sleep 1 continue fi break doneend scriptTo be honest, this is not what I create for all our MySQL servers. Instead I used this to create a chef template, chef is what we use for configuration management here (see http://www.opscode.com/ for more on chef), and here it is put to good ude to generate an Upstart script for MySQL. The above is just an example./Karlsson

Installing Lighttpd With PHP5 And MySQL Support On Ubuntu 10.10

Nov 30, 2010

Installing Lighttpd With PHP5 And MySQL Support On Ubuntu 10.10

Lighttpd is a secure, fast, standards-compliant web server designed
for speed-critical environments. This tutorial shows how you can install
Lighttpd on an Ubuntu 10.10 server with PHP5 support (through FastCGI)
and MySQL support.

Doug Schaefer: Historic Day for Linux Desktop?

Nov 05, 2010

I just tweeted this, but it’s worthy of a blog entry because I think this will, or at least should be, marked as an historic day for Linux. Mark Shuttleworth, Ubuntu chief, blogged yesterday that they are starting a transition from an X Windows based environment to the Wayland display server. That is huge news and a huge push for the fledgling Wayland project which is starting to get a lot of love lately. Intel, who employs the main developer for Wayland, already seems committed to getting MeeGo on top of it, but this move for Ubuntu all but assures Wayland will become mainstream for the Linux desktop. And I can’t wait for that!

So what the heck is Wayland and why am I so excited about it? Well I’ve been working with X Windows since my university days when X11 was spanking new. It had a great architecture that allowed the display to be hosted on a different machine than where the application ran. Back in the early 90’s that was pretty important since workstations weren’t very powerful so we still had big iron Unix servers where we ran things and being able to display them on any machine in the lab was liberating. It was the best, back in the early 90’s that is.

Then entered personal computers thanks to Microsoft Windows and to some extent Apple Macintosh. As these machines grew faster and faster, it became more economical to run your applications locally. Not only that, but the graphic architecture, where display handling was part of the operating system, allowed for the desktop environments to become rich, to the point now where we have the beautiful environments of Windows 7 and Mac OS X.

Now when Linux came along, the powers that be chose X Windows as the underlying display architecture. It made sense since X Windows is open source and it does a good job. But it is shackled by the underlying architecture that made it popular in the 90’s. As Mark put it, “I understand that it?s *possible* to get amazing results with X, but it?s extremely hard, and isn?t going to get easier. Some of the core goals of X make it harder to achieve these user experiences on X than on native GL, we?re choosing to prioritize the quality of experience over those original values, like network transparency.”

And that’s where Wayland comes it. Wikipedia describes it as “a lightweight display server for the Linux desktop. Started by Kristian H?gsberg, one of Intel OSTC member, the software’s stated goal is ‘every frame is perfect, by which I mean that applications will be able to control the rendering enough that we’ll never see tearing, lag, redrawing or flicker’”. It gives the application and window managers full control over how their content is displayed and gives them free access to the graphic hardware acceleration through OpenGL and OpenGL ES, essentially the same architecture which gives Windows and Mac their great environments.

It’s going to take some time as the ecosystem grabs hold of the possibilities. It is almost certain that other Linux distributions will jump on the bandwagon, and I’m sure nVidia and AMD will do the same with their hardware drivers. But once they do, I am convinced that this will finally make Linux a real contender in the desktop space. I can’t wait :) .

[Bash] Performing array intersection with Bash

Oct 26, 2010

I am currently working on a project to deploy new website builds to a
small number of servers. I needed something simple and reliable that could
be built in a very short period of time. I decided to whip something up in
bash with the intent of refining it in Python later.
As I began to write this code, I realized that it probably would have been
quicker to do it in Python from the start. I decided to stick with bash as
somewhat of an academic exercise. The vast majority of these deployment
scripts were trivial; check the code out of git, create a manifest, package
it up, spew it to the servers, etc, etc. The problem came during the last
step. We decided to use a symlink to point to the active build out of a
number of builds that could be available on the server at any given time.
Since all of our servers should be running the exact same version of the
build, it makes sense that I should only allow a user of my deployment
scripts to link a build which exists on all servers. But how do you
accomplish this in bash?
In most other languages, you have access to numerous array helping
functions that allow you to perform intersects, uniqs, and merges. My goal
was to do the same thing in bash without forking out to any external
binary. So how do you ensure that a particular thing exists on N number of
servers? Here it is:
function in_array() {
local x
ENTRY=$1
shift 1
ARRAY=( “$@” )
[ -z "${ARRAY}" ] && return 1
[ -z "${ENTRY}" ] && return 1
for x in ${ARRAY[@]}; do
[ "${x}" == "${ENTRY}" ] && return 0
done
return 1
}

MASTER=()
CURRENT=()
FIRST=1
for SERVER in ${SERVERS}; do
# collect all builds from server and populate CURRENT list
COMMAND=”${LS} -1fd ${WEBROOT}/${SITE}.*”
BUILDS=`${SSH} ${SSHOPTS} root@${SERVER} “${COMMAND}”`
for BUILD in ${BUILDS}; do
CURRENT=( ${CURRENT[@]-} ${BUILD} )
done

# if this is our first time around, copy CURRENT to MASTER
if [ ${FIRST} -eq 1 ]; then
MASTER=( ${CURRENT[@]} )
FIRST=0
fi

# now we do a compare between MASTER and CURRENT to see what builds
# are common
INTERSECT=()
for ENTRY in ${CURRENT[@]}; do
in_array “${ENTRY}” “${MASTER[@]}”
RET=$?
if [ "${RET}" -eq 0 ]; then
INTERSECT=( ${INTERSECT[@]-} ${ENTRY} )
fi
done
MASTER=( ${INTERSECT[@]} )

# clear the CURRENT array
CURRENT=()
done
Let me take a moment to explain the code above:

In order to check for array intersection, you need an in_array()
function

The first argument as the “needle” and the second is the
“haystack”
We verify that both parameters were passed
We simply loop through the haystack checking for the needle
If we find it, return success. Otherwise, eventually return
false

We need to loop through each server eventually, but we’ll start with
the first one

Run an SSH command to get a listing of builds
Populate an array ($CURRENT) with the builds that were found

Since the first server has no previous server to compare with, so we
just copy it to $MASTER

We then loop to the 2nd server, and put the result of getting builds
into $CURRENT

Now that we have the first server’s builds in $MASTER, we perform an
intersect with $CURRENT
We realize the need for an $INTERSECT array to hold the intersections
found above
$INTERSECT becomes $MASTER since it only contains similar builds from
the 1st and 2nd server

Looping to the 3rd server, we get the builds and put them in $CURRENT

Since $MASTER contains only the similar builds thus far, we again
compare it with $CURRENT
The intersect can now be used to compare against builds on the 4th
server, and so on

Once you finish looping through all servers, your $MASTER should
contain only similar builds

There are a few guides out there which show you how to do this via
forking, but I thought someone may appreciate the elegance of using 100%
bash to accomplish this. I hope this helps someone else out there!

InnoDB memory allocation, ulimit, and OpenSUSE

Aug 23, 2010

I recently encountered an interesting case. A customer reported that mysqld crashed on start on OpenSUSE 11.2 kernel 2.6.31.12-0.2-desktop x86_64 ? with 96 GB RAM when the innodb_buffer_pool_size was set to anything more than 62 GB. I decided to try it with 76 GB. The error message was an assert due to a failed malloc() in ut_malloc_low() in ut/ut0mem.c inside InnoDB source code. InnoDB wraps the majority of its memory allocations in ut_malloc_low(), so to get an idea of the pattern of requested allocations I added a debugging fprintf() to tell me how much was being allocated and whether it was successful.
I discovered something interesting. I expected the allocation to fail on the 76 GB of the buffer pool, due to some weird memory mapping issue and a continuous block of 76 GB not being available. However, that is not what happened. 76 GB buffer was allocated successfully. What was failing is the allocation of 3.37GB after that. What in the world could InnoDB need that was 3.37 GB? There was nothing in the settings that asked for anything close to 3 GB explicitly.
Source code is the ultimate documentation, and I took advantage of that. My good friend GDB guided me to buf_pool_init() in buf/buf0buf.c. There I found the following:
buf_pool->frame_mem = os_mem_alloc_large(
UNIV_PAGE_SIZE * (n_frames + 1),
TRUE, FALSE);
That was the buffer pool itself, the 76 GB of it. And now the buffer pool’s friend:
buf_pool->blocks = ut_malloc(sizeof(buf_block_t) * max_size);
3.6 GB of it!
From the comments in the code (InnoDB code actually has very good comments), max_size is the maximum number of buffer pool pages (16K each), n_frames which is the same thing unless AWE is used, but it was not used, so I did not worry about it.
What shall we call that friend? It is used for storing some meta information about buffer pool pages. The most natural name I could come up with from reading the source code is the blocks array.
Thus we can see that we are allocating another chunk that is in proportion to the setting of innodb_buffer_pool_size for the blocks array. The exact proportions will probably vary from version to version, but roughly about 1 G for every 25 G of the buffer pool. This can become significant in the proper innodb_buffer_pool_size estimations when the system has a lot of RAM and you want to have the largest possible innodb_buffer_pool_size. Do not forget to give the blocks array some room!
While this was an interesting investigation, it nevertheless did not explain why there was not enough room for a 76 GB buffer pool. Even with the extra 3.37 GB allocation, there was still some free memory. Or was there? Maybe some hidden monster was eating it up? I quickly wrote this hack to prove or disprove the monster’s presence.
I verified that I could allocate and initialize two chunks of 40 GB from two separate processes, but not 80 GB from one. In fact, 80GB allocation failed right in malloc(), did not even get to initialization. I tested it with allocating 70 GB concurrently in each process so as to overrun physical memory + swap. Both allocations were successful, one initialized successfully, the other was killed by the OOM kill during initialization.
This smelled like a low ulimit, and sure enough it was. ulimit -m ulimited; ulimit -v unlimited did the magic, and mysqld successfully started with an 80 GB buffer pool. Apparenly OpenSUSE defaults are set in proportion to physical memory to keep the memory-hungry applications from taking the system down. On this particular system (96 GB physical memory, 2 GB swap it decided to set the virtual memory ulimit (-v) to 77.27 GB, and the physical memory (-m) to 80.40 GB).

Entry posted by sasha |
No comment
Add to: | | | |

PHP for Android, PHP 6 canceled, APC in PHP 5.4

Jul 26, 2010

By Manuel Lemos
On this episode of the Lately in PHP podcast, Manuel Lemos and Ernani Joppert comment on the launch of the PHP for Android project and the consequences for the PHP market.

They also talk about the cancellation of PHP 6 and the inclusion of features planned for PHP 6 in PHP 5.4, like the integration of the APC cache extension in the main PHP distribution bundle.

Some of the most interesting classes nominated for the May edition of the PHP Programming Innovation Award are commented, like the PDF text extract, PHP duplicate files finder, Fast Fourier Transform and splx_graph.

linux.conf.au 2011 CFP Open!

Jul 15, 2010

Head on over to http://lca2011.linux.org.au/ and check it out!
You’ve got until August 7th to put in a paper, miniconf, poster or tutorial.
Things I’d like to see come from my kinda world:

topics on running large numbers of machines
latest in large scale web infrastructure
latest going on in the IO space: (SSD, filesystems, SSD as L2 cache)
Applications of above technologies and what it means for application performance
Scalable and massive tcp daemons (i.e. Eric should come talk on scalestack)
exploration of pain points in current technologies and discussion on ways to fix them (from people really in the know)
A Hydra tutorial: starting with stock Ubuntu lucid, and exiting the tutorial with some analysis running on my project.
Something that completely takes me off guard and is awesome.

I’d love to see people from the MySQL, Drizzle and Rackspace worlds have a decent presence. For those who’ve never heard of/been to an LCA before: we reject at least another whole conference worth of papers. It’s the conference on the calendar that everything else moves around.

Share this on Facebook

Tweet This!

Share this on del.icio.us

Digg this!

Post on Google Buzz

Koneksi TelkomselFlash dengan Modem HSDPA HUAWEI E1550 di OpenSUSE 11.2

Jul 01, 2010

Modem HSDPA Huawei E150 Pada Distro Linux OpenSUSE 11.2 , HSDPA Modem Huawey E1550? sudah Dapat dikenali dengan baik, Tentu Hal Ini akan membuat pekerjaan kita akan menjadi lebih mudah. Yang Perlu Kita Lakukan adalah tinggal mengkoneksikan modem tersebut ke Internet Provider, Pada Kasus? ini Saya akan Mengkoneksikan Modem Dengan? Telkomsel Flash, dan konfigurasinya sangat [...]